builder: fx-team-linux64-asan slave: bld-linux64-spot-076 starttime: 1457022569.82 results: retry (5) buildid: 20160303080855 builduid: a674476a281e4e268063b0adafe906f2 revision: 4e811a988416d26049193abb91acf67bc9c1fe0a ========= Started set props: master (results: 0, elapsed: 0 secs) (at 2016-03-03 08:29:29.823365) ========= master: http://buildbot-master77.bb.releng.use1.mozilla.com:8001/ ========= Finished set props: master (results: 0, elapsed: 0 secs) (at 2016-03-03 08:29:29.823764) ========= ========= Started set props: basedir (results: 0, elapsed: 0 secs) (at 2016-03-03 08:29:29.824056) ========= bash -c pwd in dir /builds/slave/fx-team-l64-asan-0000000000000/. (timeout 1200 secs) watching logfiles {} argv: ['bash', '-c', 'pwd'] environment: CCACHE_HASHDIR= G_BROKEN_FILENAMES=1 HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-spot-076.build.releng.use1.mozilla.com LANG=en_US.UTF-8 LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin PWD=/builds/slave/fx-team-l64-asan-0000000000000 SHELL=/bin/bash SHLVL=1 TERM=linux TMOUT=86400 USER=cltbld _=/tools/buildbot/bin/python using PTY: False /builds/slave/fx-team-l64-asan-0000000000000 program finished with exit code 0 elapsedTime=0.004140 basedir: '/builds/slave/fx-team-l64-asan-0000000000000' ========= master_lag: 0.15 ========= ========= Finished set props: basedir (results: 0, elapsed: 0 secs) (at 2016-03-03 08:29:29.979291) ========= ========= Started downloading to buildprops.json (results: 0, elapsed: 0 secs) (at 2016-03-03 08:29:29.979594) ========= ========= Finished downloading to buildprops.json (results: 0, elapsed: 0 secs) (at 2016-03-03 08:29:29.993126) ========= ========= Started 'rm -rf ...' (results: 0, elapsed: 0 secs) (at 2016-03-03 08:29:29.993410) ========= rm -rf properties in dir /builds/slave/fx-team-l64-asan-0000000000000/. (timeout 1200 secs) watching logfiles {} argv: ['rm', '-rf', 'properties'] environment: CCACHE_HASHDIR= G_BROKEN_FILENAMES=1 HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-spot-076.build.releng.use1.mozilla.com LANG=en_US.UTF-8 LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin PWD=/builds/slave/fx-team-l64-asan-0000000000000 SHELL=/bin/bash SHLVL=1 TERM=linux TMOUT=86400 USER=cltbld _=/tools/buildbot/bin/python using PTY: False program finished with exit code 0 elapsedTime=0.002652 ========= master_lag: 0.04 ========= ========= Finished 'rm -rf ...' (results: 0, elapsed: 0 secs) (at 2016-03-03 08:29:30.041005) ========= ========= Started set props: script_repo_url (results: 0, elapsed: 0 secs) (at 2016-03-03 08:29:30.041291) ========= script_repo_url: https://hg.mozilla.org/build/mozharness ========= Finished set props: script_repo_url (results: 0, elapsed: 0 secs) (at 2016-03-03 08:29:30.041621) ========= ========= Started 'rm -rf ...' (results: 0, elapsed: 0 secs) (at 2016-03-03 08:29:30.041895) ========= rm -rf scripts in dir /builds/slave/fx-team-l64-asan-0000000000000/. (timeout 1200 secs) watching logfiles {} argv: ['rm', '-rf', 'scripts'] environment: CCACHE_HASHDIR= G_BROKEN_FILENAMES=1 HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-spot-076.build.releng.use1.mozilla.com LANG=en_US.UTF-8 LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin PWD=/builds/slave/fx-team-l64-asan-0000000000000 SHELL=/bin/bash SHLVL=1 TERM=linux TMOUT=86400 USER=cltbld _=/tools/buildbot/bin/python using PTY: False program finished with exit code 0 elapsedTime=0.002497 ========= master_lag: 0.04 ========= ========= Finished 'rm -rf ...' (results: 0, elapsed: 0 secs) (at 2016-03-03 08:29:30.088998) ========= ========= Started 'bash -c ...' (results: 0, elapsed: 2 secs) (at 2016-03-03 08:29:30.089331) ========= bash -c 'python /tools/checkouts/build-tools/buildfarm/utils/archiver_client.py mozharness --repo integration/fx-team --rev 4e811a988416d26049193abb91acf67bc9c1fe0a --destination scripts --debug' in dir /builds/slave/fx-team-l64-asan-0000000000000/. (timeout 1200 secs) watching logfiles {} argv: ['bash', '-c', u'python /tools/checkouts/build-tools/buildfarm/utils/archiver_client.py mozharness --repo integration/fx-team --rev 4e811a988416d26049193abb91acf67bc9c1fe0a --destination scripts --debug'] environment: CCACHE_HASHDIR= G_BROKEN_FILENAMES=1 HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-spot-076.build.releng.use1.mozilla.com LANG=en_US.UTF-8 LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin PWD=/builds/slave/fx-team-l64-asan-0000000000000 SHELL=/bin/bash SHLVL=1 TERM=linux TMOUT=86400 USER=cltbld _=/tools/buildbot/bin/python using PTY: False 2016-03-03 08:29:30,126 truncating revision to first 12 chars 2016-03-03 08:29:30,126 Setting DEBUG logging. 2016-03-03 08:29:30,126 attempt 1/10 2016-03-03 08:29:30,126 Getting archive location from https://api.pub.build.mozilla.org/archiver/hgmo/integration/fx-team/4e811a988416?&preferred_region=us-west-2&suffix=tar.gz&subdir=testing/mozharness 2016-03-03 08:29:31,799 unpacking tar archive at: fx-team-4e811a988416/testing/mozharness/ program finished with exit code 0 elapsedTime=2.372820 ========= master_lag: 0.01 ========= ========= Finished 'bash -c ...' (results: 0, elapsed: 2 secs) (at 2016-03-03 08:29:32.475029) ========= ========= Started set props: script_repo_revision (results: 0, elapsed: 0 secs) (at 2016-03-03 08:29:32.475414) ========= echo 4e811a988416d26049193abb91acf67bc9c1fe0a in dir /builds/slave/fx-team-l64-asan-0000000000000/. (timeout 1200 secs) watching logfiles {} argv: ['echo', u'4e811a988416d26049193abb91acf67bc9c1fe0a'] environment: CCACHE_HASHDIR= G_BROKEN_FILENAMES=1 HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-spot-076.build.releng.use1.mozilla.com LANG=en_US.UTF-8 LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin PWD=/builds/slave/fx-team-l64-asan-0000000000000 SHELL=/bin/bash SHLVL=1 TERM=linux TMOUT=86400 USER=cltbld _=/tools/buildbot/bin/python using PTY: False 4e811a988416d26049193abb91acf67bc9c1fe0a program finished with exit code 0 elapsedTime=0.003075 script_repo_revision: '4e811a988416d26049193abb91acf67bc9c1fe0a' ========= master_lag: 0.05 ========= ========= Finished set props: script_repo_revision (results: 0, elapsed: 0 secs) (at 2016-03-03 08:29:32.524149) ========= ========= Started downloading to oauth.txt (results: 0, elapsed: 0 secs) (at 2016-03-03 08:29:32.524471) ========= ========= Finished downloading to oauth.txt (results: 0, elapsed: 0 secs) (at 2016-03-03 08:29:32.536924) ========= ========= Started '/tools/buildbot/bin/python scripts/scripts/fx_desktop_build.py ...' interrupted (results: 5, elapsed: 11 mins, 19 secs) (at 2016-03-03 08:29:32.537193) ========= /tools/buildbot/bin/python scripts/scripts/fx_desktop_build.py --config builds/releng_base_linux_64_builds.py --custom-build-variant-cfg asan --config balrog/production.py --branch fx-team --build-pool production in dir /builds/slave/fx-team-l64-asan-0000000000000/. (timeout 10800 secs) (maxTime 19800 secs) watching logfiles {} argv: ['/tools/buildbot/bin/python', 'scripts/scripts/fx_desktop_build.py', '--config', 'builds/releng_base_linux_64_builds.py', '--custom-build-variant-cfg', 'asan', '--config', 'balrog/production.py', '--branch', 'fx-team', '--build-pool', 'production'] environment: CCACHE_COMPRESS=1 CCACHE_DIR=/builds/ccache CCACHE_HASHDIR= CCACHE_UMASK=002 DISPLAY=:2 G_BROKEN_FILENAMES=1 HG_SHARE_BASE_DIR=/builds/hg-shared HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-spot-076.build.releng.use1.mozilla.com LANG=en_US.UTF-8 LC_ALL=C LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld MOZ_AUTOMATION=1 MOZ_CRASHREPORTER_NO_REPORT=1 MOZ_OBJDIR=obj-firefox MOZ_SYMBOLS_EXTRA_BUILDID=linux64-asan PATH=/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PROPERTIES_FILE=/builds/slave/fx-team-l64-asan-0000000000000/buildprops.json PWD=/builds/slave/fx-team-l64-asan-0000000000000 SHELL=/bin/bash SHLVL=1 SYMBOL_SERVER_HOST=symbolpush.mozilla.org SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_ffx/ SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/ffxbld_rsa SYMBOL_SERVER_USER=ffxbld TERM=linux TINDERBOX_OUTPUT=1 TMOUT=86400 TOOLTOOL_CACHE=/builds/tooltool_cache TOOLTOOL_HOME=/builds USER=cltbld _=/tools/buildbot/bin/python using PTY: False 08:29:32 INFO - MultiFileLogger online at 20160303 08:29:32 in /builds/slave/fx-team-l64-asan-0000000000000 08:29:32 INFO - Script was run with '--custom-build-variant-cfg asan' and 'asan' matches a key in 'builds/releng_sub_linux_configs/64_asan.py'. Updating self.config with items from that key's value. 08:29:32 INFO - Script was run with '--branch fx-team' and 'fx-team' matches a key in 'builds/branch_specifics.py'. Updating self.config with items from that key's value. 08:29:32 INFO - Script was run with '--build-pool production' and 'production' matches a key in 'builds/build_pool_specifics.py'. Updating self.config with items from that key's value. 08:29:32 INFO - To generate a config file based upon options passed and config files used, run script as before but extend options with "--dump-config" 08:29:32 INFO - For a diff of where self.config got its items, run the script again as before but extend options with: "--dump-config-hierarchy" 08:29:32 INFO - Both --dump-config and --dump-config-hierarchy don't actually run any actions. 08:29:32 INFO - We are running this in buildbot, grab the build props 08:29:32 INFO - Using buildbot properties: 08:29:32 INFO - { 08:29:32 INFO - "project": "", 08:29:32 INFO - "product": "firefox", 08:29:32 INFO - "script_repo_revision": "production", 08:29:32 INFO - "scheduler": "fx-team-firefox", 08:29:32 INFO - "repository": "", 08:29:32 INFO - "buildername": "Linux x86-64 fx-team asan build", 08:29:32 INFO - "buildid": "20160303080855", 08:29:32 INFO - "basedir": "/builds/slave/fx-team-l64-asan-0000000000000", 08:29:32 INFO - "buildnumber": 6, 08:29:32 INFO - "platform": "linux64-asan", 08:29:32 INFO - "master": "http://buildbot-master77.bb.releng.use1.mozilla.com:8001/", 08:29:32 INFO - "branch": "fx-team", 08:29:32 INFO - "revision": "4e811a988416d26049193abb91acf67bc9c1fe0a", 08:29:32 INFO - "slavename": "bld-linux64-spot-076", 08:29:32 INFO - "builduid": "a674476a281e4e268063b0adafe906f2", 08:29:32 INFO - "repo_path": "integration/fx-team" 08:29:32 INFO - } 08:29:32 INFO - Setting buildbot property stage_platform to linux64-asan 08:29:32 INFO - mkdir: /builds/slave/fx-team-l64-asan-0000000000000/properties 08:29:32 INFO - Writing buildbot properties ['stage_platform'] to /builds/slave/fx-team-l64-asan-0000000000000/properties/stage_platform 08:29:32 INFO - Writing to file /builds/slave/fx-team-l64-asan-0000000000000/properties/stage_platform 08:29:32 INFO - Contents: 08:29:32 INFO - stage_platform:linux64-asan 08:29:32 INFO - Run as scripts/scripts/fx_desktop_build.py --config builds/releng_base_linux_64_builds.py --custom-build-variant-cfg asan --config balrog/production.py --branch fx-team --build-pool production 08:29:32 INFO - Determining buildid from buildbot properties 08:29:32 INFO - Setting buildbot property buildid to 20160303080855 08:29:32 INFO - Writing buildbot properties ['buildid'] to /builds/slave/fx-team-l64-asan-0000000000000/properties/buildid 08:29:32 INFO - Writing to file /builds/slave/fx-team-l64-asan-0000000000000/properties/buildid 08:29:32 INFO - Contents: 08:29:32 INFO - buildid:20160303080855 08:29:32 INFO - Determining builduid from buildbot properties 08:29:32 INFO - Setting buildbot property builduid to a674476a281e4e268063b0adafe906f2 08:29:32 INFO - Writing buildbot properties ['builduid'] to /builds/slave/fx-team-l64-asan-0000000000000/properties/builduid 08:29:32 INFO - Writing to file /builds/slave/fx-team-l64-asan-0000000000000/properties/builduid 08:29:32 INFO - Contents: 08:29:32 INFO - builduid:a674476a281e4e268063b0adafe906f2 08:29:32 INFO - Creating virtualenv /builds/slave/fx-team-l64-asan-0000000000000/build/venv 08:29:32 INFO - mkdir: /builds/slave/fx-team-l64-asan-0000000000000/build 08:29:32 INFO - Running command: ['virtualenv', '--no-site-packages', '--distribute', '/builds/slave/fx-team-l64-asan-0000000000000/build/venv'] in /builds/slave/fx-team-l64-asan-0000000000000/build 08:29:32 INFO - Copy/paste: virtualenv --no-site-packages --distribute /builds/slave/fx-team-l64-asan-0000000000000/build/venv 08:29:32 INFO - The --no-site-packages flag is deprecated; it is now the default behavior. 08:29:32 INFO - New python executable in /builds/slave/fx-team-l64-asan-0000000000000/build/venv/bin/python2.7 08:29:32 INFO - Also creating executable in /builds/slave/fx-team-l64-asan-0000000000000/build/venv/bin/python 08:29:33 INFO - Installing distribute.............................................................................................................................................................................................done. 08:29:33 INFO - Installing pip...............done. 08:29:33 INFO - Return code: 0 08:29:33 INFO - Installing requests==2.8.1 into virtualenv /builds/slave/fx-team-l64-asan-0000000000000/build/venv 08:29:33 INFO - retry: Calling run_command with args: [['/builds/slave/fx-team-l64-asan-0000000000000/build/venv/bin/pip', 'install', '--timeout', '120', 'requests==2.8.1']], kwargs: {'error_level': 'warning', 'error_list': [{'substr': 'not found or a compiler error:', 'level': 'warning'}, {'regex': <_sre.SRE_Pattern object at 0x1c0a030>, 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x1b1bc00>, 'level': 'warning'}, {'regex': <_sre.SRE_Pattern object at 0x1dc4d10>, 'level': 'debug'}, {'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x1b08370>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x1b16540>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x1d73350>, 'level': 'critical'}], 'cwd': '/builds/slave/fx-team-l64-asan-0000000000000/build', 'env': {'MOZ_AUTOMATION': '1', 'LESSOPEN': '|/usr/bin/lesspipe.sh %s', 'TMOUT': '86400', 'LOGNAME': 'cltbld', 'USER': 'cltbld', 'MOZ_OBJDIR': 'obj-firefox', 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 'SYMBOL_SERVER_USER': 'ffxbld', 'DISPLAY': ':2', 'CCACHE_UMASK': '002', 'LANG': 'en_US.UTF-8', 'CCACHE_HASHDIR': '', 'TERM': 'linux', 'SHELL': '/bin/bash', 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-asan', 'SHLVL': '1', 'G_BROKEN_FILENAMES': '1', 'HISTSIZE': '1000', 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 'HOME': '/home/cltbld', 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 'CCACHE_DIR': '/builds/ccache', 'TINDERBOX_OUTPUT': '1', 'MAIL': '/var/spool/mail/cltbld', 'LC_ALL': 'C', 'TOOLTOOL_HOME': '/builds', '_': '/tools/buildbot/bin/python', 'HOSTNAME': 'bld-linux64-spot-076.build.releng.use1.mozilla.com', 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_rsa', 'MOZBUILD_STATE_PATH': '/builds/slave/fx-team-l64-asan-0000000000000/.mozbuild', 'HISTCONTROL': 'ignoredups', 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 'PWD': '/builds/slave/fx-team-l64-asan-0000000000000', 'PROPERTIES_FILE': '/builds/slave/fx-team-l64-asan-0000000000000/buildprops.json', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'CCACHE_COMPRESS': '1', 'TOOLTOOL_CACHE': '/builds/tooltool_cache'}}, attempt #1 08:29:33 INFO - Running command: ['/builds/slave/fx-team-l64-asan-0000000000000/build/venv/bin/pip', 'install', '--timeout', '120', 'requests==2.8.1'] in /builds/slave/fx-team-l64-asan-0000000000000/build 08:29:33 INFO - Copy/paste: /builds/slave/fx-team-l64-asan-0000000000000/build/venv/bin/pip install --timeout 120 requests==2.8.1 08:29:33 INFO - Using env: {'CCACHE_COMPRESS': '1', 08:29:33 INFO - 'CCACHE_DIR': '/builds/ccache', 08:29:33 INFO - 'CCACHE_HASHDIR': '', 08:29:33 INFO - 'CCACHE_UMASK': '002', 08:29:33 INFO - 'DISPLAY': ':2', 08:29:33 INFO - 'G_BROKEN_FILENAMES': '1', 08:29:33 INFO - 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 08:29:33 INFO - 'HISTCONTROL': 'ignoredups', 08:29:33 INFO - 'HISTSIZE': '1000', 08:29:33 INFO - 'HOME': '/home/cltbld', 08:29:33 INFO - 'HOSTNAME': 'bld-linux64-spot-076.build.releng.use1.mozilla.com', 08:29:33 INFO - 'LANG': 'en_US.UTF-8', 08:29:33 INFO - 'LC_ALL': 'C', 08:29:33 INFO - 'LESSOPEN': '|/usr/bin/lesspipe.sh %s', 08:29:33 INFO - 'LOGNAME': 'cltbld', 08:29:33 INFO - 'MAIL': '/var/spool/mail/cltbld', 08:29:33 INFO - 'MOZBUILD_STATE_PATH': '/builds/slave/fx-team-l64-asan-0000000000000/.mozbuild', 08:29:33 INFO - 'MOZ_AUTOMATION': '1', 08:29:33 INFO - 'MOZ_CRASHREPORTER_NO_REPORT': '1', 08:29:33 INFO - 'MOZ_OBJDIR': 'obj-firefox', 08:29:33 INFO - 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-asan', 08:29:33 INFO - 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 08:29:33 INFO - 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 08:29:33 INFO - 'PROPERTIES_FILE': '/builds/slave/fx-team-l64-asan-0000000000000/buildprops.json', 08:29:33 INFO - 'PWD': '/builds/slave/fx-team-l64-asan-0000000000000', 08:29:33 INFO - 'SHELL': '/bin/bash', 08:29:33 INFO - 'SHLVL': '1', 08:29:33 INFO - 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 08:29:33 INFO - 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 08:29:33 INFO - 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_rsa', 08:29:33 INFO - 'SYMBOL_SERVER_USER': 'ffxbld', 08:29:33 INFO - 'TERM': 'linux', 08:29:33 INFO - 'TINDERBOX_OUTPUT': '1', 08:29:33 INFO - 'TMOUT': '86400', 08:29:33 INFO - 'TOOLTOOL_CACHE': '/builds/tooltool_cache', 08:29:33 INFO - 'TOOLTOOL_HOME': '/builds', 08:29:33 INFO - 'USER': 'cltbld', 08:29:33 INFO - '_': '/tools/buildbot/bin/python'} 08:29:36 INFO - Ignoring indexes: http://pypi.python.org/simple/ 08:29:36 INFO - Downloading/unpacking requests==2.8.1 08:29:36 INFO - Running setup.py egg_info for package requests 08:29:36 INFO - Installing collected packages: requests 08:29:36 INFO - Running setup.py install for requests 08:29:36 INFO - Successfully installed requests 08:29:36 INFO - Cleaning up... 08:29:36 INFO - Return code: 0 08:29:36 INFO - Installing PyHawk-with-a-single-extra-commit==0.1.5 into virtualenv /builds/slave/fx-team-l64-asan-0000000000000/build/venv 08:29:36 INFO - retry: Calling run_command with args: [['/builds/slave/fx-team-l64-asan-0000000000000/build/venv/bin/pip', 'install', '--timeout', '120', 'PyHawk-with-a-single-extra-commit==0.1.5']], kwargs: {'error_level': 'warning', 'error_list': [{'substr': 'not found or a compiler error:', 'level': 'warning'}, {'regex': <_sre.SRE_Pattern object at 0x1c0a030>, 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x1b1bc00>, 'level': 'warning'}, {'regex': <_sre.SRE_Pattern object at 0x1dc4d10>, 'level': 'debug'}, {'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x1b08370>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x1b16540>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x1d73350>, 'level': 'critical'}], 'cwd': '/builds/slave/fx-team-l64-asan-0000000000000/build', 'env': {'MOZ_AUTOMATION': '1', 'LESSOPEN': '|/usr/bin/lesspipe.sh %s', 'TMOUT': '86400', 'LOGNAME': 'cltbld', 'USER': 'cltbld', 'MOZ_OBJDIR': 'obj-firefox', 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 'SYMBOL_SERVER_USER': 'ffxbld', 'DISPLAY': ':2', 'CCACHE_UMASK': '002', 'LANG': 'en_US.UTF-8', 'CCACHE_HASHDIR': '', 'TERM': 'linux', 'SHELL': '/bin/bash', 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-asan', 'SHLVL': '1', 'G_BROKEN_FILENAMES': '1', 'HISTSIZE': '1000', 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 'HOME': '/home/cltbld', 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 'CCACHE_DIR': '/builds/ccache', 'TINDERBOX_OUTPUT': '1', 'MAIL': '/var/spool/mail/cltbld', 'LC_ALL': 'C', 'TOOLTOOL_HOME': '/builds', '_': '/tools/buildbot/bin/python', 'HOSTNAME': 'bld-linux64-spot-076.build.releng.use1.mozilla.com', 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_rsa', 'MOZBUILD_STATE_PATH': '/builds/slave/fx-team-l64-asan-0000000000000/.mozbuild', 'HISTCONTROL': 'ignoredups', 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 'PWD': '/builds/slave/fx-team-l64-asan-0000000000000', 'PROPERTIES_FILE': '/builds/slave/fx-team-l64-asan-0000000000000/buildprops.json', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'CCACHE_COMPRESS': '1', 'TOOLTOOL_CACHE': '/builds/tooltool_cache'}}, attempt #1 08:29:36 INFO - Running command: ['/builds/slave/fx-team-l64-asan-0000000000000/build/venv/bin/pip', 'install', '--timeout', '120', 'PyHawk-with-a-single-extra-commit==0.1.5'] in /builds/slave/fx-team-l64-asan-0000000000000/build 08:29:36 INFO - Copy/paste: /builds/slave/fx-team-l64-asan-0000000000000/build/venv/bin/pip install --timeout 120 PyHawk-with-a-single-extra-commit==0.1.5 08:29:36 INFO - Using env: {'CCACHE_COMPRESS': '1', 08:29:36 INFO - 'CCACHE_DIR': '/builds/ccache', 08:29:36 INFO - 'CCACHE_HASHDIR': '', 08:29:36 INFO - 'CCACHE_UMASK': '002', 08:29:36 INFO - 'DISPLAY': ':2', 08:29:36 INFO - 'G_BROKEN_FILENAMES': '1', 08:29:36 INFO - 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 08:29:36 INFO - 'HISTCONTROL': 'ignoredups', 08:29:36 INFO - 'HISTSIZE': '1000', 08:29:36 INFO - 'HOME': '/home/cltbld', 08:29:36 INFO - 'HOSTNAME': 'bld-linux64-spot-076.build.releng.use1.mozilla.com', 08:29:36 INFO - 'LANG': 'en_US.UTF-8', 08:29:36 INFO - 'LC_ALL': 'C', 08:29:36 INFO - 'LESSOPEN': '|/usr/bin/lesspipe.sh %s', 08:29:36 INFO - 'LOGNAME': 'cltbld', 08:29:36 INFO - 'MAIL': '/var/spool/mail/cltbld', 08:29:36 INFO - 'MOZBUILD_STATE_PATH': '/builds/slave/fx-team-l64-asan-0000000000000/.mozbuild', 08:29:36 INFO - 'MOZ_AUTOMATION': '1', 08:29:36 INFO - 'MOZ_CRASHREPORTER_NO_REPORT': '1', 08:29:36 INFO - 'MOZ_OBJDIR': 'obj-firefox', 08:29:36 INFO - 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-asan', 08:29:36 INFO - 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 08:29:36 INFO - 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 08:29:36 INFO - 'PROPERTIES_FILE': '/builds/slave/fx-team-l64-asan-0000000000000/buildprops.json', 08:29:36 INFO - 'PWD': '/builds/slave/fx-team-l64-asan-0000000000000', 08:29:36 INFO - 'SHELL': '/bin/bash', 08:29:36 INFO - 'SHLVL': '1', 08:29:36 INFO - 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 08:29:36 INFO - 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 08:29:36 INFO - 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_rsa', 08:29:36 INFO - 'SYMBOL_SERVER_USER': 'ffxbld', 08:29:36 INFO - 'TERM': 'linux', 08:29:36 INFO - 'TINDERBOX_OUTPUT': '1', 08:29:36 INFO - 'TMOUT': '86400', 08:29:36 INFO - 'TOOLTOOL_CACHE': '/builds/tooltool_cache', 08:29:36 INFO - 'TOOLTOOL_HOME': '/builds', 08:29:36 INFO - 'USER': 'cltbld', 08:29:36 INFO - '_': '/tools/buildbot/bin/python'} 08:29:37 INFO - Ignoring indexes: http://pypi.python.org/simple/ 08:29:37 INFO - Downloading/unpacking PyHawk-with-a-single-extra-commit==0.1.5 08:29:37 INFO - Downloading PyHawk-with-a-single-extra-commit-0.1.5.tar.gz 08:29:37 INFO - Running setup.py egg_info for package PyHawk-with-a-single-extra-commit 08:29:37 INFO - Installing collected packages: PyHawk-with-a-single-extra-commit 08:29:37 INFO - Running setup.py install for PyHawk-with-a-single-extra-commit 08:29:37 INFO - Successfully installed PyHawk-with-a-single-extra-commit 08:29:37 INFO - Cleaning up... 08:29:37 INFO - Return code: 0 08:29:37 INFO - Installing taskcluster==0.0.26 into virtualenv /builds/slave/fx-team-l64-asan-0000000000000/build/venv 08:29:37 INFO - retry: Calling run_command with args: [['/builds/slave/fx-team-l64-asan-0000000000000/build/venv/bin/pip', 'install', '--timeout', '120', 'taskcluster==0.0.26']], kwargs: {'error_level': 'warning', 'error_list': [{'substr': 'not found or a compiler error:', 'level': 'warning'}, {'regex': <_sre.SRE_Pattern object at 0x1c0a030>, 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x1b1bc00>, 'level': 'warning'}, {'regex': <_sre.SRE_Pattern object at 0x1dc4d10>, 'level': 'debug'}, {'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x1b08370>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x1b16540>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x1d73350>, 'level': 'critical'}], 'cwd': '/builds/slave/fx-team-l64-asan-0000000000000/build', 'env': {'MOZ_AUTOMATION': '1', 'LESSOPEN': '|/usr/bin/lesspipe.sh %s', 'TMOUT': '86400', 'LOGNAME': 'cltbld', 'USER': 'cltbld', 'MOZ_OBJDIR': 'obj-firefox', 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 'SYMBOL_SERVER_USER': 'ffxbld', 'DISPLAY': ':2', 'CCACHE_UMASK': '002', 'LANG': 'en_US.UTF-8', 'CCACHE_HASHDIR': '', 'TERM': 'linux', 'SHELL': '/bin/bash', 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-asan', 'SHLVL': '1', 'G_BROKEN_FILENAMES': '1', 'HISTSIZE': '1000', 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 'HOME': '/home/cltbld', 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 'CCACHE_DIR': '/builds/ccache', 'TINDERBOX_OUTPUT': '1', 'MAIL': '/var/spool/mail/cltbld', 'LC_ALL': 'C', 'TOOLTOOL_HOME': '/builds', '_': '/tools/buildbot/bin/python', 'HOSTNAME': 'bld-linux64-spot-076.build.releng.use1.mozilla.com', 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_rsa', 'MOZBUILD_STATE_PATH': '/builds/slave/fx-team-l64-asan-0000000000000/.mozbuild', 'HISTCONTROL': 'ignoredups', 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 'PWD': '/builds/slave/fx-team-l64-asan-0000000000000', 'PROPERTIES_FILE': '/builds/slave/fx-team-l64-asan-0000000000000/buildprops.json', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'CCACHE_COMPRESS': '1', 'TOOLTOOL_CACHE': '/builds/tooltool_cache'}}, attempt #1 08:29:37 INFO - Running command: ['/builds/slave/fx-team-l64-asan-0000000000000/build/venv/bin/pip', 'install', '--timeout', '120', 'taskcluster==0.0.26'] in /builds/slave/fx-team-l64-asan-0000000000000/build 08:29:37 INFO - Copy/paste: /builds/slave/fx-team-l64-asan-0000000000000/build/venv/bin/pip install --timeout 120 taskcluster==0.0.26 08:29:37 INFO - Using env: {'CCACHE_COMPRESS': '1', 08:29:37 INFO - 'CCACHE_DIR': '/builds/ccache', 08:29:37 INFO - 'CCACHE_HASHDIR': '', 08:29:37 INFO - 'CCACHE_UMASK': '002', 08:29:37 INFO - 'DISPLAY': ':2', 08:29:37 INFO - 'G_BROKEN_FILENAMES': '1', 08:29:37 INFO - 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 08:29:37 INFO - 'HISTCONTROL': 'ignoredups', 08:29:37 INFO - 'HISTSIZE': '1000', 08:29:37 INFO - 'HOME': '/home/cltbld', 08:29:37 INFO - 'HOSTNAME': 'bld-linux64-spot-076.build.releng.use1.mozilla.com', 08:29:37 INFO - 'LANG': 'en_US.UTF-8', 08:29:37 INFO - 'LC_ALL': 'C', 08:29:37 INFO - 'LESSOPEN': '|/usr/bin/lesspipe.sh %s', 08:29:37 INFO - 'LOGNAME': 'cltbld', 08:29:37 INFO - 'MAIL': '/var/spool/mail/cltbld', 08:29:37 INFO - 'MOZBUILD_STATE_PATH': '/builds/slave/fx-team-l64-asan-0000000000000/.mozbuild', 08:29:37 INFO - 'MOZ_AUTOMATION': '1', 08:29:37 INFO - 'MOZ_CRASHREPORTER_NO_REPORT': '1', 08:29:37 INFO - 'MOZ_OBJDIR': 'obj-firefox', 08:29:37 INFO - 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-asan', 08:29:37 INFO - 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 08:29:37 INFO - 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 08:29:37 INFO - 'PROPERTIES_FILE': '/builds/slave/fx-team-l64-asan-0000000000000/buildprops.json', 08:29:37 INFO - 'PWD': '/builds/slave/fx-team-l64-asan-0000000000000', 08:29:37 INFO - 'SHELL': '/bin/bash', 08:29:37 INFO - 'SHLVL': '1', 08:29:37 INFO - 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 08:29:37 INFO - 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 08:29:37 INFO - 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_rsa', 08:29:37 INFO - 'SYMBOL_SERVER_USER': 'ffxbld', 08:29:37 INFO - 'TERM': 'linux', 08:29:37 INFO - 'TINDERBOX_OUTPUT': '1', 08:29:37 INFO - 'TMOUT': '86400', 08:29:37 INFO - 'TOOLTOOL_CACHE': '/builds/tooltool_cache', 08:29:37 INFO - 'TOOLTOOL_HOME': '/builds', 08:29:37 INFO - 'USER': 'cltbld', 08:29:37 INFO - '_': '/tools/buildbot/bin/python'} 08:29:40 INFO - Ignoring indexes: http://pypi.python.org/simple/ 08:29:40 INFO - Downloading/unpacking taskcluster==0.0.26 08:29:40 INFO - Downloading taskcluster-0.0.26.tar.gz 08:29:40 INFO - Running setup.py egg_info for package taskcluster 08:29:40 INFO - Downloading/unpacking requests>=2.4.3,<=2.7.0 (from taskcluster==0.0.26) 08:29:40 INFO - Running setup.py egg_info for package requests 08:29:40 INFO - Requirement already satisfied (use --upgrade to upgrade): PyHawk-with-a-single-extra-commit==0.1.5 in ./venv/lib/python2.7/site-packages (from taskcluster==0.0.26) 08:29:40 INFO - Downloading/unpacking slugid (from taskcluster==0.0.26) 08:29:40 INFO - Downloading slugid-1.0.7.tar.gz 08:29:40 INFO - Running setup.py egg_info for package slugid 08:29:40 INFO - Installing collected packages: taskcluster, requests, slugid 08:29:40 INFO - Running setup.py install for taskcluster 08:29:40 INFO - Found existing installation: requests 2.8.1 08:29:40 INFO - Uninstalling requests: 08:29:40 INFO - Successfully uninstalled requests 08:29:40 INFO - Running setup.py install for requests 08:29:40 INFO - Running setup.py install for slugid 08:29:40 INFO - Successfully installed taskcluster requests slugid 08:29:40 INFO - Cleaning up... 08:29:40 INFO - Return code: 0 08:29:40 INFO - Done creating virtualenv /builds/slave/fx-team-l64-asan-0000000000000/build/venv. 08:29:40 INFO - Getting output from command: ['/builds/slave/fx-team-l64-asan-0000000000000/build/venv/bin/pip', 'freeze'] 08:29:40 INFO - Copy/paste: /builds/slave/fx-team-l64-asan-0000000000000/build/venv/bin/pip freeze 08:29:40 INFO - Reading from file tmpfile_stdout 08:29:40 INFO - Current package versions: 08:29:40 INFO - PyHawk-with-a-single-extra-commit == 0.1.5 08:29:40 INFO - distribute == 0.6.24 08:29:40 INFO - requests == 2.7.0 08:29:40 INFO - slugid == 1.0.7 08:29:40 INFO - taskcluster == 0.0.26 08:29:40 INFO - wsgiref == 0.1.2 08:29:40 INFO - Running pre-run listener: influxdb_recording_init 08:29:40 INFO - build_resources.json path: /builds/slave/fx-team-l64-asan-0000000000000/build/src/obj-firefox/.mozbuild/build_resources.json 08:29:40 INFO - rmtree: /builds/slave/fx-team-l64-asan-0000000000000/build/src/obj-firefox/.mozbuild/build_resources.json 08:29:40 INFO - Getting output from command: ['/builds/slave/fx-team-l64-asan-0000000000000/build/venv/bin/python', '-c', 'from distutils.sysconfig import get_python_lib; print(get_python_lib())'] 08:29:40 INFO - Copy/paste: /builds/slave/fx-team-l64-asan-0000000000000/build/venv/bin/python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())" 08:29:41 INFO - Reading from file tmpfile_stdout 08:29:41 INFO - Output received: 08:29:41 INFO - /builds/slave/fx-team-l64-asan-0000000000000/build/venv/lib/python2.7/site-packages 08:29:41 INFO - Dumping config to /builds/slave/fx-team-l64-asan-0000000000000/logs/localconfig.json. 08:29:41 INFO - {'app_ini_path': '%(obj_dir)s/dist/bin/application.ini', 08:29:41 INFO - 'append_to_log': False, 08:29:41 INFO - 'balrog_credentials_file': 'oauth.txt', 08:29:41 INFO - 'balrog_servers': ({'balrog_api_root': 'https://aus4-admin.mozilla.org/api', 08:29:41 INFO - 'balrog_usernames': {'Fennec': 'ffxbld', 08:29:41 INFO - 'b2g': 'b2gbld', 08:29:41 INFO - 'firefox': 'ffxbld', 08:29:41 INFO - 'graphene': 'ffxbld', 08:29:41 INFO - 'horizon': 'ffxbld', 08:29:41 INFO - 'mobile': 'ffxbld', 08:29:41 INFO - 'thunderbird': 'tbirdbld'}, 08:29:41 INFO - 'ignore_failures': False, 08:29:41 INFO - 'url_replacements': (('http://archive.mozilla.org/pub', 08:29:41 INFO - 'http://download.cdn.mozilla.net/pub'),)}, 08:29:41 INFO - {'balrog_api_root': 'https://aus4-admin-dev.allizom.org/api', 08:29:41 INFO - 'balrog_usernames': {'Fennec': 'stage-ffxbld', 08:29:41 INFO - 'b2g': 'stage-b2gbld', 08:29:41 INFO - 'firefox': 'stage-ffxbld', 08:29:41 INFO - 'graphene': 'stage-ffxbld', 08:29:41 INFO - 'horizon': 'stage-ffxbld', 08:29:41 INFO - 'mobile': 'stage-ffxbld', 08:29:41 INFO - 'thunderbird': 'stage-tbirdbld'}, 08:29:41 INFO - 'ignore_failures': True}), 08:29:41 INFO - 'base_name': 'Linux_x86-64_%(branch)s', 08:29:41 INFO - 'base_work_dir': '/builds/slave/fx-team-l64-asan-0000000000000', 08:29:41 INFO - 'branch': 'fx-team', 08:29:41 INFO - 'build_pool': 'production', 08:29:41 INFO - 'build_resources_path': '%(abs_src_dir)s/obj-firefox/.mozbuild/build_resources.json', 08:29:41 INFO - 'build_type': 'asan', 08:29:41 INFO - 'build_variant': 'asan', 08:29:41 INFO - 'buildbot_json_path': 'buildprops.json', 08:29:41 INFO - 'check_test_env': {'MINIDUMP_SAVE_PATH': '%(base_work_dir)s/minidumps', 08:29:41 INFO - 'MINIDUMP_STACKWALK': '%(abs_tools_dir)s/breakpad/linux64/minidump_stackwalk'}, 08:29:41 INFO - 'clobberer_url': 'https://api.pub.build.mozilla.org/clobberer/lastclobber', 08:29:41 INFO - 'clone_by_revision': False, 08:29:41 INFO - 'clone_with_purge': False, 08:29:41 INFO - 'compare_locales_repo': 'https://hg.mozilla.org/build/compare-locales', 08:29:41 INFO - 'compare_locales_rev': 'RELEASE_AUTOMATION', 08:29:41 INFO - 'compare_locales_vcs': 'hgtool', 08:29:41 INFO - 'config_files': ('builds/releng_base_linux_64_builds.py', 08:29:41 INFO - '/builds/slave/fx-team-l64-asan-0000000000000/scripts/scripts/../configs/builds/releng_sub_linux_configs/64_asan.py', 08:29:41 INFO - 'balrog/production.py', 08:29:41 INFO - 'builds/branch_specifics.py', 08:29:41 INFO - 'builds/build_pool_specifics.py'), 08:29:41 INFO - 'debug_build': False, 08:29:41 INFO - 'default_actions': ('clobber', 08:29:41 INFO - 'clone-tools', 08:29:41 INFO - 'checkout-sources', 08:29:41 INFO - 'setup-mock', 08:29:41 INFO - 'build', 08:29:41 INFO - 'upload-files', 08:29:41 INFO - 'sendchange', 08:29:41 INFO - 'check-test'), 08:29:41 INFO - 'default_vcs': 'hgtool', 08:29:41 INFO - 'enable_ccache': True, 08:29:41 INFO - 'enable_count_ctors': True, 08:29:41 INFO - 'enable_signing': False, 08:29:41 INFO - 'enable_talos_sendchange': False, 08:29:41 INFO - 'enable_unittest_sendchange': True, 08:29:41 INFO - 'env': {'CCACHE_COMPRESS': '1', 08:29:41 INFO - 'CCACHE_DIR': '/builds/ccache', 08:29:41 INFO - 'CCACHE_UMASK': '002', 08:29:41 INFO - 'DISPLAY': ':2', 08:29:41 INFO - 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 08:29:41 INFO - 'LC_ALL': 'C', 08:29:41 INFO - 'MOZBUILD_STATE_PATH': '/builds/slave/fx-team-l64-asan-0000000000000/.mozbuild', 08:29:41 INFO - 'MOZ_AUTOMATION': '1', 08:29:41 INFO - 'MOZ_CRASHREPORTER_NO_REPORT': '1', 08:29:41 INFO - 'MOZ_OBJDIR': 'obj-firefox', 08:29:41 INFO - 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 08:29:41 INFO - 'TINDERBOX_OUTPUT': '1', 08:29:41 INFO - 'TOOLTOOL_CACHE': '/builds/tooltool_cache', 08:29:41 INFO - 'TOOLTOOL_HOME': '/builds'}, 08:29:41 INFO - 'exes': {'buildbot': '/tools/buildbot/bin/buildbot', 08:29:41 INFO - 'hgtool.py': '/builds/slave/fx-team-l64-asan-0000000000000/build/tools/buildfarm/utils/hgtool.py'}, 08:29:41 INFO - 'graph_selector': '/server/collect.cgi', 08:29:41 INFO - 'graph_server': 'graphs.mozilla.org', 08:29:41 INFO - 'hgtool_base_bundle_urls': ('https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles',), 08:29:41 INFO - 'include_post_upload_builddir': False, 08:29:41 INFO - 'influx_credentials_file': 'oauth.txt', 08:29:41 INFO - 'is_automation': True, 08:29:41 INFO - 'latest_mar_dir': '/pub/mozilla.org/firefox/nightly/latest-%(branch)s', 08:29:41 INFO - 'log_level': 'info', 08:29:41 INFO - 'log_to_console': True, 08:29:41 INFO - 'mock_files': (('/home/cltbld/.ssh', '/home/mock_mozilla/.ssh'), 08:29:41 INFO - ('/home/cltbld/.hgrc', '/builds/.hgrc'), 08:29:41 INFO - ('/home/cltbld/.boto', '/builds/.boto'), 08:29:41 INFO - ('/builds/gapi.data', '/builds/gapi.data'), 08:29:41 INFO - ('/builds/relengapi.tok', '/builds/relengapi.tok'), 08:29:41 INFO - ('/tools/tooltool.py', '/builds/tooltool.py'), 08:29:41 INFO - ('/builds/mozilla-desktop-geoloc-api.key', 08:29:41 INFO - '/builds/mozilla-desktop-geoloc-api.key'), 08:29:41 INFO - ('/builds/crash-stats-api.token', 08:29:41 INFO - '/builds/crash-stats-api.token'), 08:29:41 INFO - ('/builds/adjust-sdk.token', '/builds/adjust-sdk.token'), 08:29:41 INFO - ('/builds/adjust-sdk-beta.token', 08:29:41 INFO - '/builds/adjust-sdk-beta.token'), 08:29:41 INFO - ('/usr/local/lib/hgext', '/usr/local/lib/hgext')), 08:29:41 INFO - 'mock_mozilla_dir': '/builds/mock_mozilla', 08:29:41 INFO - 'mock_packages': ('autoconf213', 08:29:41 INFO - 'python', 08:29:41 INFO - 'mozilla-python27', 08:29:41 INFO - 'zip', 08:29:41 INFO - 'mozilla-python27-mercurial', 08:29:41 INFO - 'git', 08:29:41 INFO - 'ccache', 08:29:41 INFO - 'perl-Test-Simple', 08:29:41 INFO - 'perl-Config-General', 08:29:41 INFO - 'yasm', 08:29:41 INFO - 'wget', 08:29:41 INFO - 'mpfr', 08:29:41 INFO - 'xorg-x11-font*', 08:29:41 INFO - 'imake', 08:29:41 INFO - 'gcc45_0moz3', 08:29:41 INFO - 'gcc454_0moz1', 08:29:41 INFO - 'gcc472_0moz1', 08:29:41 INFO - 'gcc473_0moz1', 08:29:41 INFO - 'yasm', 08:29:41 INFO - 'ccache', 08:29:41 INFO - 'valgrind', 08:29:41 INFO - 'dbus-x11', 08:29:41 INFO - 'glibc-static', 08:29:41 INFO - 'libstdc++-static', 08:29:41 INFO - 'gtk2-devel', 08:29:41 INFO - 'libnotify-devel', 08:29:41 INFO - 'alsa-lib-devel', 08:29:41 INFO - 'libcurl-devel', 08:29:41 INFO - 'wireless-tools-devel', 08:29:41 INFO - 'libX11-devel', 08:29:41 INFO - 'libXt-devel', 08:29:41 INFO - 'mesa-libGL-devel', 08:29:41 INFO - 'gnome-vfs2-devel', 08:29:41 INFO - 'GConf2-devel', 08:29:41 INFO - 'gcc45_0moz3', 08:29:41 INFO - 'gcc454_0moz1', 08:29:41 INFO - 'gcc472_0moz1', 08:29:41 INFO - 'gcc473_0moz1', 08:29:41 INFO - 'yasm', 08:29:41 INFO - 'ccache', 08:29:41 INFO - 'pulseaudio-libs-devel', 08:29:41 INFO - 'gstreamer-devel', 08:29:41 INFO - 'gstreamer-plugins-base-devel', 08:29:41 INFO - 'freetype-2.3.11-6.el6_1.8.x86_64', 08:29:41 INFO - 'freetype-devel-2.3.11-6.el6_1.8.x86_64'), 08:29:41 INFO - 'mock_target': 'mozilla-centos6-x86_64', 08:29:41 INFO - 'nightly_build': False, 08:29:41 INFO - 'nightly_promotion_branches': ('mozilla-central', 'mozilla-aurora'), 08:29:41 INFO - 'objdir': 'obj-firefox', 08:29:41 INFO - 'old_packages': ('%(objdir)s/dist/firefox-*', 08:29:41 INFO - '%(objdir)s/dist/fennec*', 08:29:41 INFO - '%(objdir)s/dist/seamonkey*', 08:29:41 INFO - '%(objdir)s/dist/thunderbird*', 08:29:41 INFO - '%(objdir)s/dist/install/sea/*.exe'), 08:29:41 INFO - 'opt_config_files': (), 08:29:41 INFO - 'periodic_clobber': 168, 08:29:41 INFO - 'pgo_build': False, 08:29:41 INFO - 'pgo_platforms': ('linux', 'linux64', 'win32', 'win64'), 08:29:41 INFO - 'platform': 'linux64', 08:29:41 INFO - 'platform_supports_post_upload_to_latest': False, 08:29:41 INFO - 'publish_nightly_en_US_routes': False, 08:29:41 INFO - 'release_to_try_builds': False, 08:29:41 INFO - 'repo_base': 'https://hg.mozilla.org', 08:29:41 INFO - 'repo_path': 'integration/fx-team', 08:29:41 INFO - 'sendchange_masters': ('buildbot-master81.build.mozilla.org:9301',), 08:29:41 INFO - 'src_mozconfig': 'browser/config/mozconfigs/linux64/nightly-asan', 08:29:41 INFO - 'stage_platform': 'linux64-asan', 08:29:41 INFO - 'stage_product': 'firefox', 08:29:41 INFO - 'stage_server': 'upload.ffxbld.productdelivery.prod.mozaws.net', 08:29:41 INFO - 'stage_ssh_key': 'ffxbld_rsa', 08:29:41 INFO - 'stage_username': 'ffxbld', 08:29:41 INFO - 'symbol_server_host': 'symbolpush.mozilla.org', 08:29:41 INFO - 'taskcluster_credentials_file': 'oauth.txt', 08:29:41 INFO - 'taskcluster_index': 'index', 08:29:41 INFO - 'tinderbox_build_dir': None, 08:29:41 INFO - 'to_tinderbox_dated': True, 08:29:41 INFO - 'tools_repo': 'https://hg.mozilla.org/build/tools', 08:29:41 INFO - 'tooltool_bootstrap': 'setup.sh', 08:29:41 INFO - 'tooltool_manifest_src': 'browser/config/tooltool-manifests/linux64/asan.manifest', 08:29:41 INFO - 'tooltool_script': ('/builds/tooltool.py',), 08:29:41 INFO - 'tooltool_url': 'https://api.pub.build.mozilla.org/tooltool/', 08:29:41 INFO - 'upload_env': {'UPLOAD_HOST': '%(stage_server)s', 08:29:41 INFO - 'UPLOAD_SSH_KEY': '/home/mock_mozilla/.ssh/%(stage_ssh_key)s', 08:29:41 INFO - 'UPLOAD_TO_TEMP': '1', 08:29:41 INFO - 'UPLOAD_USER': '%(stage_username)s'}, 08:29:41 INFO - 'use_branch_in_symbols_extra_buildid': True, 08:29:41 INFO - 'use_clobberer': True, 08:29:41 INFO - 'use_platform_in_symbols_extra_buildid': True, 08:29:41 INFO - 'vcs_share_base': '/builds/hg-shared', 08:29:41 INFO - 'virtualenv_modules': ('requests==2.8.1', 08:29:41 INFO - 'PyHawk-with-a-single-extra-commit==0.1.5', 08:29:41 INFO - 'taskcluster==0.0.26'), 08:29:41 INFO - 'virtualenv_path': 'venv', 08:29:41 INFO - 'volatile_config': {'actions': None, 'add_actions': None, 'no_actions': None}, 08:29:41 INFO - 'who': '', 08:29:41 INFO - 'work_dir': 'build'} 08:29:41 INFO - ##### 08:29:41 INFO - ##### Running clobber step. 08:29:41 INFO - ##### 08:29:41 INFO - Running pre-action listener: influxdb_recording_pre_action 08:29:41 INFO - Running main action method: clobber 08:29:41 INFO - Running post-action listener: influxdb_recording_post_action 08:29:41 INFO - Starting new HTTPS connection (1): goldiewilson-onepointtwentyone-1.c.influxdb.com /builds/slave/fx-team-l64-asan-0000000000000/build/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning 08:29:41 INFO - ##### 08:29:41 INFO - ##### Running clone-tools step. 08:29:41 INFO - ##### 08:29:41 INFO - Running pre-action listener: influxdb_recording_pre_action 08:29:41 INFO - Running main action method: clone_tools 08:29:41 INFO - retry: Calling _get_revision with args: (, '/builds/slave/fx-team-l64-asan-0000000000000/build/tools'), kwargs: {}, attempt #1 08:29:41 INFO - Setting /builds/slave/fx-team-l64-asan-0000000000000/build/tools to https://hg.mozilla.org/build/tools using shared directory /builds/hg-shared. 08:29:41 INFO - Checking if share extension works. 08:29:41 INFO - Getting output from command: ['hg', '--config', 'ui.merge=internal:merge', 'help', 'share'] 08:29:41 INFO - Copy/paste: hg --config ui.merge=internal:merge help share 08:29:41 INFO - Reading from file tmpfile_stdout 08:29:41 INFO - hg share works. 08:29:41 INFO - Updating shared repo 08:29:41 INFO - Pulling https://hg.mozilla.org/build/tools to /builds/hg-shared/build/tools and updating. 08:29:41 INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'pull', 'https://hg.mozilla.org/build/tools'] in /builds/hg-shared/build/tools 08:29:41 INFO - Copy/paste: hg --config ui.merge=internal:merge pull https://hg.mozilla.org/build/tools 08:29:41 INFO - Calling ['hg', '--config', 'ui.merge=internal:merge', 'pull', 'https://hg.mozilla.org/build/tools'] with output_timeout 1200 08:29:42 INFO - pulling from https://hg.mozilla.org/build/tools 08:29:42 INFO - searching for changes 08:29:42 INFO - no changes found 08:29:42 INFO - Return code: 0 08:29:42 INFO - Updating /builds/hg-shared/build/tools. 08:29:42 INFO - Getting output from command: ['hg', '--config', 'ui.merge=internal:merge', 'branch'] in /builds/hg-shared/build/tools 08:29:42 INFO - Copy/paste: hg --config ui.merge=internal:merge branch 08:29:43 INFO - Reading from file tmpfile_stdout 08:29:43 INFO - Output received: 08:29:43 INFO - default 08:29:43 INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'update', '-C'] in /builds/hg-shared/build/tools 08:29:43 INFO - Copy/paste: hg --config ui.merge=internal:merge update -C 08:29:44 INFO - 0 files updated, 0 files merged, 0 files removed, 0 files unresolved 08:29:44 INFO - Return code: 0 08:29:44 INFO - Getting output from command: ['hg', '--config', 'ui.merge=internal:merge', 'parent', '--template', '{node}'] in /builds/hg-shared/build/tools 08:29:44 INFO - Copy/paste: hg --config ui.merge=internal:merge parent --template {node} 08:29:44 INFO - Reading from file tmpfile_stdout 08:29:44 INFO - Output received: 08:29:44 INFO - 3753f5815c837a3594c883b72a13e3d956413fb8 08:29:44 INFO - Trying to share /builds/hg-shared/build/tools to /builds/slave/fx-team-l64-asan-0000000000000/build/tools 08:29:44 INFO - Sharing /builds/hg-shared/build/tools to /builds/slave/fx-team-l64-asan-0000000000000/build/tools. 08:29:44 INFO - mkdir: /builds/slave/fx-team-l64-asan-0000000000000/build/tools 08:29:44 INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'share', '-U', '/builds/hg-shared/build/tools', '/builds/slave/fx-team-l64-asan-0000000000000/build/tools'] 08:29:44 INFO - Copy/paste: hg --config ui.merge=internal:merge share -U /builds/hg-shared/build/tools /builds/slave/fx-team-l64-asan-0000000000000/build/tools 08:29:45 INFO - Return code: 0 08:29:45 INFO - Updating /builds/slave/fx-team-l64-asan-0000000000000/build/tools. 08:29:45 INFO - Getting output from command: ['hg', '--config', 'ui.merge=internal:merge', 'branch'] in /builds/slave/fx-team-l64-asan-0000000000000/build/tools 08:29:45 INFO - Copy/paste: hg --config ui.merge=internal:merge branch 08:29:45 INFO - Reading from file tmpfile_stdout 08:29:45 INFO - Output received: 08:29:45 INFO - default 08:29:45 INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'update', '-C'] in /builds/slave/fx-team-l64-asan-0000000000000/build/tools 08:29:45 INFO - Copy/paste: hg --config ui.merge=internal:merge update -C 08:29:46 INFO - 1165 files updated, 0 files merged, 0 files removed, 0 files unresolved 08:29:46 INFO - Return code: 0 08:29:46 INFO - Getting output from command: ['hg', '--config', 'ui.merge=internal:merge', 'parent', '--template', '{node}'] in /builds/slave/fx-team-l64-asan-0000000000000/build/tools 08:29:46 INFO - Copy/paste: hg --config ui.merge=internal:merge parent --template {node} 08:29:47 INFO - Reading from file tmpfile_stdout 08:29:47 INFO - Output received: 08:29:47 INFO - 3753f5815c837a3594c883b72a13e3d956413fb8 08:29:47 INFO - Running post-action listener: influxdb_recording_post_action 08:29:47 INFO - Resetting dropped connection: goldiewilson-onepointtwentyone-1.c.influxdb.com /builds/slave/fx-team-l64-asan-0000000000000/build/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning 08:29:47 INFO - ##### 08:29:47 INFO - ##### Running checkout-sources step. 08:29:47 INFO - ##### 08:29:47 INFO - Running pre-action listener: influxdb_recording_pre_action 08:29:47 INFO - Running main action method: checkout_sources 08:29:47 INFO - retry: Calling _get_revision with args: (, '/builds/slave/fx-team-l64-asan-0000000000000/build/src'), kwargs: {}, attempt #1 08:29:47 INFO - Running command: ['/builds/slave/fx-team-l64-asan-0000000000000/build/tools/buildfarm/utils/hgtool.py', '-r', '4e811a988416d26049193abb91acf67bc9c1fe0a', '--bundle', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/fx-team.hg', 'https://hg.mozilla.org/integration/fx-team', '/builds/slave/fx-team-l64-asan-0000000000000/build/src'] 08:29:47 INFO - Copy/paste: /builds/slave/fx-team-l64-asan-0000000000000/build/tools/buildfarm/utils/hgtool.py -r 4e811a988416d26049193abb91acf67bc9c1fe0a --bundle https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/fx-team.hg https://hg.mozilla.org/integration/fx-team /builds/slave/fx-team-l64-asan-0000000000000/build/src 08:29:47 INFO - Using env: {'CCACHE_COMPRESS': '1', 08:29:47 INFO - 'CCACHE_DIR': '/builds/ccache', 08:29:47 INFO - 'CCACHE_HASHDIR': '', 08:29:47 INFO - 'CCACHE_UMASK': '002', 08:29:47 INFO - 'DISPLAY': ':2', 08:29:47 INFO - 'G_BROKEN_FILENAMES': '1', 08:29:47 INFO - 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 08:29:47 INFO - 'HISTCONTROL': 'ignoredups', 08:29:47 INFO - 'HISTSIZE': '1000', 08:29:47 INFO - 'HOME': '/home/cltbld', 08:29:47 INFO - 'HOSTNAME': 'bld-linux64-spot-076.build.releng.use1.mozilla.com', 08:29:47 INFO - 'LANG': 'en_US.UTF-8', 08:29:47 INFO - 'LC_ALL': 'C', 08:29:47 INFO - 'LESSOPEN': '|/usr/bin/lesspipe.sh %s', 08:29:47 INFO - 'LOGNAME': 'cltbld', 08:29:47 INFO - 'MAIL': '/var/spool/mail/cltbld', 08:29:47 INFO - 'MOZBUILD_STATE_PATH': '/builds/slave/fx-team-l64-asan-0000000000000/.mozbuild', 08:29:47 INFO - 'MOZ_AUTOMATION': '1', 08:29:47 INFO - 'MOZ_BUILD_DATE': '20160303080855', 08:29:47 INFO - 'MOZ_CRASHREPORTER_NO_REPORT': '1', 08:29:47 INFO - 'MOZ_OBJDIR': 'obj-firefox', 08:29:47 INFO - 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-asan', 08:29:47 INFO - 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 08:29:47 INFO - 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 08:29:47 INFO - 'PROPERTIES_FILE': '/builds/slave/fx-team-l64-asan-0000000000000/buildprops.json', 08:29:47 INFO - 'PWD': '/builds/slave/fx-team-l64-asan-0000000000000', 08:29:47 INFO - 'SHELL': '/bin/bash', 08:29:47 INFO - 'SHLVL': '1', 08:29:47 INFO - 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 08:29:47 INFO - 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 08:29:47 INFO - 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_rsa', 08:29:47 INFO - 'SYMBOL_SERVER_USER': 'ffxbld', 08:29:47 INFO - 'TERM': 'linux', 08:29:47 INFO - 'TINDERBOX_OUTPUT': '1', 08:29:47 INFO - 'TMOUT': '86400', 08:29:47 INFO - 'TOOLTOOL_CACHE': '/builds/tooltool_cache', 08:29:47 INFO - 'TOOLTOOL_HOME': '/builds', 08:29:47 INFO - 'USER': 'cltbld', 08:29:47 INFO - '_': '/tools/buildbot/bin/python'} 08:29:47 INFO - Reporting hg version in use 08:29:47 INFO - command: START 08:29:47 INFO - command: hg -q version 08:29:47 INFO - command: cwd: . 08:29:47 INFO - command: output: 08:29:47 INFO - Mercurial Distributed SCM (version 3.2.1) 08:29:47 INFO - command: END (0.30s elapsed) 08:29:47 INFO - Checking if share extension works 08:29:47 INFO - command: START 08:29:47 INFO - command: hg help share 08:29:47 INFO - command: cwd: /builds/slave/fx-team-l64-asan-0000000000000 08:29:47 INFO - command: env: {'HGPLAIN': '1'} 08:29:48 INFO - command: END (0.50s elapsed) 08:29:48 INFO - Updating shared repo 08:29:48 INFO - Reporting hg version in use 08:29:48 INFO - command: START 08:29:48 INFO - command: hg -q version 08:29:48 INFO - command: cwd: . 08:29:48 INFO - command: output: 08:29:48 INFO - Mercurial Distributed SCM (version 3.2.1) 08:29:48 INFO - command: END (0.30s elapsed) 08:29:48 INFO - Attempting to initialize clone with bundles 08:29:48 INFO - command: START 08:29:48 INFO - command: hg init /builds/hg-shared/integration/fx-team 08:29:48 INFO - command: cwd: /builds/slave/fx-team-l64-asan-0000000000000 08:29:48 INFO - command: output: 08:29:49 INFO - command: END (0.32s elapsed) 08:29:49 INFO - Trying to use bundle https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/fx-team.hg 08:29:49 INFO - command: START 08:29:49 INFO - command: hg unbundle --traceback https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/fx-team.hg 08:29:49 INFO - command: cwd: /builds/hg-shared/integration/fx-team 08:29:49 INFO - command: env: {'HGPLAIN': '1'} remoteFailed: [Failure instance: Traceback (failure with no frames): : Connection to the other side was lost in a non-clean fashion. ] [Failure instance: Traceback (failure with no frames): : Connection to the other side was lost in a non-clean fashion. ] ========= Finished '/tools/buildbot/bin/python scripts/scripts/fx_desktop_build.py ...' interrupted (results: 5, elapsed: 11 mins, 19 secs) (at 2016-03-03 08:40:52.042521) ========= ========= Skipped (results: not started, elapsed: not started) ========= ========= Skipped (results: not started, elapsed: not started) ========= ========= Skipped (results: not started, elapsed: not started) ========= ========= Total master_lag: 0.30 =========