TechnicalOperations Status Report

Project Operations from 2017-04-01 to 2017-07-01

Help

Network Operations 164594 codfw: ganeti2007-ganeti2008 switch port configuration Screep Done None
Network Operations 150400 Spare pool servers switch configuration Screep Done None
Network Operations 116766 attach mr1-codfw to new out of band mgmt link Screep Done None
Network Operations 134932 cr2-codfw LUCHIP/trinity_pio error messages Screep Done None
Network Operations 146391 eeden ethernet outage In-Scope Open None
Network Operations 122406 Consider renumbering Labs to separate address spaces Screep Open None
Traffic 123431 Investigate if login.m.wikimedia.org needs to stay around Screep Done None
Traffic 164608 Merge cache_maps into cache_upload functionally Screep Done None
Traffic 88614 Spike: what to do about Special:RecordImpression and Banner History Screep Done 2.0
Traffic 133490 Wikidata Query Service REST endpoint returns truncated results Screep Done None
Traffic 161819 Investigate 502 errors from nginx when backend returns 302 In-Scope Done None
Traffic 147784 Mediawiki thumbnail requests for 0px should result in http 400 not 500 Screep Done None
Traffic 159574 LDF endpoint ordering is not stable between servers when paging Screep Done None
Traffic 155807 Monitor Certificate Transparency (CT) logs Screep Done None
Traffic 155504 Fix universal link support in iOS when the OS requests the site association file from m.wikipedia.org Screep Done None
Traffic 132685 Preload STS for wikimedia.org Screep Done None
Traffic 141454 piuparts fail with WARN: Broken symlinks: /etc/systemd/system... Screep Done None
Traffic 108649 Set up "w.wiki" domain for usage with UrlShortener Screep Done None
Traffic 164868 SSL error for https://wikispecies.org/ Screep Open None
Traffic 128559 store.wikimedia.org HTTPS issues Screep Open None
Traffic 165765 Refactor pybal/LVS config for shared failover Screep Open None
Traffic 164609 Merge cache_misc into cache_text functionally Screep Open None
Traffic 164579 Investigate nginx reload behavior Screep Open None
Traffic 154801 Investigate varnishd child crashes when multiple nodes get depooled/pooled concurrently In-Scope Open None
Traffic 154702 Fix broken referer categorization for visits from Safari browsers Screep Open None
Traffic 106517 upload.wikimedia.org returns HTTP status code 503 for truncated urls, not 404 Screep Open None
Traffic 152882 Many misc wikis lack mobile domains Screep Open None
Traffic 109331 Deleted files sometimes remain visible to non-privileged users if permanently linked In-Scope Open None
Traffic 109776 Tilerator should purge Varnish cache Screep Open None
Traffic 112316 Configure varnish to use "Unconfigured domain" page for 404 Not Served (instead of generic error) Screep Open None
Traffic 144508 Point wikipedia.in to 205.147.101.160 instead of URL forward Screep Open None
Traffic 119372 Pybal IdleConnectionMonitor with TCP KeepAlive shows random fails if more than 100 servers are involved. Screep Open None
Traffic 141266 letsencrypt puppetization: add parallel rsa+ecdsa cert support In-Scope Open None
Traffic 120121 Improve Varnish XFF processing for trusted proxies Screep Open None
Traffic 120486 add a https-only option to dynamicproxy Screep Open None
Traffic 75944 Monitor Varnish caches on beta cluster have two varnishd process running In-Scope Open None
Traffic 137979 Support brotli compression In-Scope Open None
DBA 120122 Perform a rolling restart of all MySQL slaves (masters too for those services with low traffic) Screep Done None
DBA 118816 mysql privs: restrict access to racktables to krypton Screep Done None
DBA 166141 db1046 BBU looks faulty Screep Done None
DBA 160731 db1048 BBU Faulty - slave lagging In-Scope Done None
DBA 130702 Several es20XX servers keep crashing (es2017, es2019, es2015, es2014) since 23 March Screep Done None
DBA 156006 Move db1052 to row B3 Screep Done None
DBA 147774 Physically move db1053 to a different rack Screep Done None
DBA 123731 Migrate labsdb1005/1006/1007 to jessie Screep Done None
DBA 119619 Decide storage backend for performance schema monitoring stats Screep Done None
DBA 127570 Rename be_x_oldwiki database to be_taraskwiki In-Scope Open None
DBA 50930 Database replication problems - production and labs (tracking) Screep Open None
DBA 119626 Eliminate SPOF at the main database infrastructure In-Scope Open None
DBA 168584 Labsdb* servers need to be rebooted Screep Open None
DBA 134476 Decommission old coredb machines (<=db1050) In-Scope Open None
DBA 141252 icinga hp raid check timeout on busy ms-be and db machines In-Scope Open None
DBA 167031 Global rename of Idh0854 → Garam: supervision needed Screep Open None
DBA 163143 dbtree: don't return 200 on error pages Screep Open None
Software Development 148494 Add shell scripts CI validations Screep Open None
Software Development 150560 More verbose messages from service-checker-swagger Screep Open None
Software Development 159045 Update Puppet repo code that uses maniphest.update and maniphest.createtask conduit api Screep Open None
Hardware Requests 158204 Eqiad: (2) hardware access request for labnet1003/1004 Screep Done None
Hardware Requests 84958 eqiad: (3) servers for logstash service Screep Done None
Hardware Requests 119698 spare swift disks order Screep Done None
Hardware Requests 120842 EQIAD/CODFW: 2 hardware access request for monitoring Screep Done None
Hardware Requests 121007 Decommission and remove from racks out of warranty spares Screep Done None
Hardware Requests 122100 wipe disks and add nembus back to server spares Screep Done None
Hardware Requests 142578 codfw/eqiad:(9+9) hardware access request for ORES In-Scope Done None
Hardware Requests 147309 Decommission db1019 In-Scope Done None
Hardware Requests 164513 reclaim tempdb2001(WMF6407) to spares Screep Done None
Other Operations 138627 Enable WDQS admins to enable/disable mask/unmask updater service Screep Done None
Other Operations 155181 codfw:mw2251-mw2260 switch port configuration Screep Done None
Other Operations 98183 labstore1002 issues while trying to reboot Screep Done None
Other Operations 139234 Phabricator XSS vulnerability Screep Done None
Other Operations 139728 Create a Goal project for technical operations Q1 goal "Modernize Puppet configuration management infrastructure" Screep Done None
Other Operations 130174 elastic1005.eqiad.wmnet is non-responsive Screep Done None
Other Operations 156154 ship R620 power supplies to ulsfo Screep Done None
Other Operations 156529 fix log reading permissions for dc-ops admin group Screep Done None
Other Operations 157483 Request for access to stat1003 for Sam Tarling Screep Done None
Other Operations 158131 hard-reset DRAC gadolinium.mgmt.eqiad.wmnet In-Scope Done None
Other Operations 158176 Build / migrate to HHVM 3.18 Screep Done None
Other Operations 47877 During deployment old servers may populate new cache URIs Screep Done None
Other Operations 158337 codfw: ms-be2028-ms-be2039 rack/setup In-Scope Done None
Other Operations 158664 elasticsearch logs are duplicated in journald In-Scope Done None
Other Operations 94822 Pybal RunCommand monitor doesn't work correctly on ubuntu trusty Screep Done None
Other Operations 159168 Add elasticsearch 5 .deb to reprepro experimental repository Screep Done None
Other Operations 140598 db2056 RAID controller (temporary) failure Screep Done None
Other Operations 159486 setup/deploy scb2005 & scb2006 Screep Done None
Other Operations 94611 Automate compiling service dependencies using production Jessie libraries Screep Done None
Other Operations 159887 rack and cable frdev1001 In-Scope Done None
Other Operations 140886 Unable to access Etherpad - https://etherpad.wikimedia.org/p/Fundraising_Staff_Feedback Screep Done None
Other Operations 140899 Move labs graphite to graphite-labs.wikimedia.org Screep Done None
Other Operations 92306 cp3011 hardware fault Screep Done None
Other Operations 127808 access for nikerabbit to researchers Screep Done None
Other Operations 161158 Degraded RAID on ocg1001 In-Scope Done None
Other Operations 141242 Make the WMF puppet tree compile equally under puppet 3.4 and 3.8 Screep Done None
Other Operations 92239 cron's puppet-run fails silently if apt-get update fails Screep Done None
Other Operations 123315 URGENT - remove jimmy@ alias from exim mail aliases Screep Done None
Other Operations 131147 ms-be2008.codfw.wmnet: slot=1 dev=sdl failed Screep Done None
Other Operations 141360 stat1004 doesn't show up in ganglia Screep Done None
Other Operations 162216 rack and set up analyics1058-1069 Screep Done None
Other Operations 118976 establish new thresholds for cassandra alarms after switching restbase to dtcs Screep Done None
Other Operations 141522 Physically decommission mw1001-mw1148 (except mw1017 and mw1099) Screep Done None
Other Operations 131150 Add monitoring metric for connection tracking table usage Screep Done None
Other Operations 163196 Puppet facts around the primary network interface and IPv4/IPv6 address Screep Done None
Other Operations 131877 setup stat1004/WMF4721 for hadoop client usage Screep Done None
Other Operations 163408 (icinga/grafana) webpagetest-alerts: slow page rendering for Internet Explorer Screep Done None
Other Operations 163889 cp1066 troubleshoot IPMI issue Screep Done None
Other Operations 163942 Ops pwstore currently read-only Screep Done None
Other Operations 143718 Request access to data for reader research Screep Done None
Other Operations 144380 Install and configure new WDQS nodes on codfw Screep Done None
Other Operations 90883 mc2004 console is unreadable remotely Screep Done None
Other Operations 89912 HHVM lock-ups Screep Done None
Other Operations 86668 Make all ldap users have a sane shell (/bin/bash) Screep Done None
Other Operations 86633 Unify ::production / ::beta roles for *oid Screep Done None
Other Operations 84044 clean up non-working otrs email addresses Screep Done None
Other Operations 144389 /a/mw-log/hhvm.log: file not found on Fluorine Screep Done None
Other Operations 165220 Degraded RAID on labstore1003 Screep Done None
Other Operations 144396 Disable cron job to clear elasticsearch caches and validate that it does not have significant impact on GC Screep Done None
Other Operations 165467 Create an etcd cluster in codfw for kubernetes usage Screep Done None
Other Operations 144416 reset password of winedale-l Screep Done None
Other Operations 165733 Supersede RT tickets references Screep Done None
Other Operations 81030 gdash reports for php/apache errors Screep Done None
Other Operations 116673 puppetize wdqs service startup on boot Screep Done None
Other Operations 132033 Backport libvpx2 to jessie Screep Done None
Other Operations 144801 Icinga check for unaccepted Salt keys Screep Done None
Other Operations 145618 Thumbor SVG regexp insufficient Screep Done None
Other Operations 145626 thumbor ffmpeg pipe deadlock Screep Done None
Other Operations 166345 wmf/1.30.0-wmf.2 performance issue for Wikipedias Screep Done None
Other Operations 166552 Switch etcd back to eqiad, document switchover procedure Screep Done None
Other Operations 166776 Geoshape and geoline subservices need monitoring Screep Done None
Other Operations 166806 Server side upload for Yann Screep Done None
Other Operations 71604 acct (process and login accounting) fill up instances /var/ partition Screep Done None
Other Operations 146158 Configure varnish to include wdqs nodes in codfw Screep Done None
Other Operations 146207 publish lag and response time for wdqs codfw to graphite Screep Done 2.0
Other Operations 167878 mwlog1001.eqiad.wmnet ‘/srv/mw-log/hhvm.log’ for reading: No such file or directory Screep Done None
Other Operations 132422 cronspam from labscontrol1001, labstore1001, labnet1002.eqiad.wmnet, labsdb1003.eqiad.wmnet Screep Done None
Other Operations 63442 Use gzip transfer encoding for SVG files from MediaWiki static assets Screep Done None
Other Operations 146464 hhvm root:adm owned log files cause failures for logrotate Screep Done None
Other Operations 123681 add datacenter-ops to dhcp /install-server and allow to run puppet commands Screep Done None
Other Operations 128057 db2018 failed disk (degraded RAID) Screep Done None
Other Operations 125061 Prepare mobileapps for the codfw switchover Screep Done None
Other Operations 132751 "Elastica: missing curl_init_pooled method" due to mwscript job running with PHP 5 on terbium Screep Done None
Other Operations 147181 Docker installation for production kubernetes Screep Done None
Other Operations 128369 ganglia cluster name validation in puppet Screep Done None
Other Operations 147397 Upgrade OTRS to 5.0.13 Screep Done None
Other Operations 116184 mw1083's sda disk is dying Screep Done None
Other Operations 115965 [Regression] 404 Not Found: https://en.wikipedia.org/apple-touch-icon.png Screep Done None
Other Operations 148361 SVG and VIPS engines expect temp folder to stay a while when it doesn't Screep Done None
Other Operations 114443 EventBus MVP Screep Done None
Other Operations 149228 Rename stat100x machines to have misc element names In-Scope Done None
Other Operations 132952 Failed to deploy WDQS Screep Done None
Other Operations 132976 rack/setup/deploy restbase200[7-9] Screep Done None
Other Operations 150316 decom gallium (data center) In-Scope Done None
Other Operations 123406 exim alias remove - feed-sales Screep Done None
Other Operations 129152 Deploy servermon with scap3 In-Scope Done None
Other Operations 129168 fr-all fails with error 451 Screep Done None
Other Operations 134241 graphoid should not use the http proxy to connect to the mediawiki api and other internal services Screep Done None
Other Operations 122062 Create the wikimania2017 wiki Screep Done None
Other Operations 122005 mw1228 reporting readonly file system Screep Done None
Other Operations 127053 iridium / phabricator not accepting email via smtp Screep Done None
Other Operations 123525 reduce amount of remaining Ubuntu 12.04 (precise) systems in production Screep Done None
Other Operations 151338 rack/setup prometheus200[3-4] Screep Done None
Other Operations 151456 Thumbor original file download limit should be 4GB Screep Done None
Other Operations 151818 update label/racktables visible label for thorium/wmf4726 Screep Done 3.0
Other Operations 152023 adrian bielefeldt & julius gonsior shell request + analytics-privatedata-users Screep Done None
Other Operations 129260 Requesting access to to analytics-search-user for Mikhail Popov and Oliver Keyes Screep Done None
Other Operations 152473 Upgrade Grafana to 4.0.2 Screep Done None
Other Operations 135557 kafka1013 hardware crash Screep Done None
Other Operations 152643 rack/setup/install wdqs1003 Screep Done None
Other Operations 129973 Dig through logs from 15 Mar 2016 read-only test and file bugs Screep Done None
Other Operations 137851 "Is maps service alive?" check Screep Done None
Other Operations 121231 restbase should send metrics in batches Screep Done None
Other Operations 101377 LDAP TLS failing on some instances due to inconsistent state Screep Done None
Other Operations 138415 eqiad: Install ssds to labmon1001 Screep Done None
Other Operations 156136 Increase swift replication factor for accounts Screep Open None
Other Operations 127488 (re)move problemsdonating aliases In-Scope Open None
Other Operations 129222 Icinga disk space check should also check inode usage Screep Open None
Other Operations 129224 on labcontrol1001, /var/cache/salt has too many files! In-Scope Open None
Other Operations 130617 Collect metrics on pool counter usage In-Scope Open None
Other Operations 122825 Service Ownership and Maintenance Screep Open None
Other Operations 132921 Unable to delete file pages on commons: MWException/LocalFileLockError: "Could not acquire lock" In-Scope Open None
Other Operations 122210 Security audit for tftp on Carbon In-Scope Open None
Other Operations 133656 Have a paging check for Nova API accessible Screep Open None
Other Operations 133913 Completely port l10nupdate to scap In-Scope Open None
Other Operations 135124 Deploy etcddump (or another etcd dump & load tool) to production Screep Open None
Other Operations 135318 Document how to handle 'inconsistent state within the internal storage backends' issues In-Scope Open None
Other Operations 135338 On Trusty and Jessie PHP yields: PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/20-xhprof.ini on line 2 Screep Open None
Other Operations 136311 Monitor the BMC's event log for hardware errors In-Scope Open None
Other Operations 140442 reinstall rdb100[56] with RAID Screep Open None
Other Operations 141038 implement icinga paging for non-ops teams In-Scope Open None
Other Operations 120165 Implement role based hiera lookups for labs In-Scope Open None
Other Operations 119846 Redirect revisions from svn.wikimedia.org to https://phabricator.wikimedia.org/rSVN In-Scope Open None
Other Operations 141756 audit / test / upgrade hp smartarray P840 firmware Screep Open None
Other Operations 168892 rack/setup/install labtestservices2002.wikimedia.org Screep Open None
Other Operations 144539 Remove /srv/deployment/wdqs/wdqs/rules.log symlink In-Scope Open None
Other Operations 116580 monitor postgresql replication status Screep Open None
Other Operations 146664 Limit resources used by ORES Screep Open None
Other Operations 146914 grain-ensure erroneous mismatch with (bool)True vs (str)true In-Scope Open None
Other Operations 169035 bast3002 sdb broken Screep Open None
Other Operations 149432 puppet compiler claims "no change" when catalogs are actually different In-Scope Open None
Other Operations 149617 Integrating MediaWiki (and other services) with dynamic configuration Screep Open None
Other Operations 113104 Set up a service IP for logstash Screep Open None
Other Operations 150460 Configure maps cluster to send statsd metrics to the statsd endpoint in the same datacenter In-Scope Open None
Other Operations 111301 pmtpa remnants in trebuchet redis Screep Open None
Other Operations 150672 Provide a /parsoid directory on releases.wikimedia.org Screep Open None
Other Operations 150771 Secondary production Jenkins for CI In-Scope Open None
Other Operations 151045 Extending Yubico 2FA for production use (meta bug) In-Scope Open None
Other Operations 151049 Run systematic availability tests Screep Open None
Other Operations 151050 Proper documentation for Yubico 2FA for production use Screep Open None
Other Operations 152439 cronspam from labtestservices2001 /etc/dns-floating-ip-updater.py > /dev/null In-Scope Open None
Other Operations 152562 Port fundraising stats off Ganglia Screep Open None
Other Operations 106937 Monitor [[Special:ListFiles]] for non 200 HTTP statuses in thumbnails In-Scope Open None
Other Operations 153416 docker-engine pulled into our repositories only keeps the latest version Screep Open None
Other Operations 153770 Error on RCStream server startup for the "flash policy server" In-Scope Open None
Other Operations 101141 udp rcvbuferrors and inerrors on graphite1001 In-Scope Open None
Other Operations 154915 Get rid of "import realm.pp" in manifests/site.pp In-Scope Open None
Other Operations 97909 Upgrade jobrunners to redis 2.8 In-Scope Open None
Other Operations 97524 ocg alarm ocg_job_status_queue 'flapping' Screep Open None
Other Operations 126221 Evaluate efficacy of DateTieredCompactionStrategy In-Scope Open None
Other Operations 159242 Segmentation fault creating thumbnail In-Scope Open None
Other Operations 160101 Upgrade php5-json .deb to at least 1.3.8 Screep Open None
Other Operations 160158 Make disabled accounts visible in the corp mirror LDAP replica In-Scope Open None
Other Operations 160941 Improve SSH access information in onboarding documentation Screep Open None
Other Operations 161528 incident 20170323-wikibase did not trigger Icinga paging Screep Open None
Other Operations 161918 videoscalers (mw1168, mw1169) - high load / overheating In-Scope Open None
Other Operations 161920 logrotate for ruthenium In-Scope Open None
Other Operations 162029 Migrate all jessie hosts to Linux 4.9 Screep Open None
Other Operations 162770 SATA errors for stat1004 in the dmesg Screep Open None
Other Operations 162949 hosts with puppet compiler failures on every run Screep Open None
Other Operations 91637 Rack ms-be3006 and ms-be3007 In-Scope Open None
Other Operations 163288 Decide on /var/lib vs /home as locations of homedir for l10nupdate Screep Open None
Other Operations 164290 Set up external DNS record for wikitech-static Screep Open None
Other Operations 169360 Unresponsive/misconfigured iDRACs over the host-BMC interface Screep Open None
Other Operations 165173 rack/setup/install dumpsdata100[12] Screep Open None
Other Operations 165348 Check long-running screen/tmux sessions Screep Open None
Other Operations 165520 rack and setup wtp1025-1048 Screep Open None
Other Operations 165781 rack/setup/install labcontrol100[34] Screep Open None
Other Operations 165784 rack/setup/install labmon1002 Screep Open None
Other Operations 166038 Sync internal nutcracker package with Debian package Screep Open None
Other Operations 76306 Set warning thresholds for average cluster utilization Screep Open None
Other Operations 166322 spam from phabricator in labs Screep Open None
Other Operations 167157 rack/setup/install labtestpuppetmaster2001 Screep Open None
Other Operations 167820 rack/setup/install labweb100[12].wikimedia.org Screep Open None
Other Operations 167992 rack/setup/install new kafka nodes kafka-jumbo100[1-6] Screep Open None
Other Operations 168619 Degraded RAID on lvs3001 Screep Open None
Other Operations 124413 confctl should provide tags information after writing data In-Scope Open None