TechnicalOperations Open Tasks

Help.

149936 Drop flow_subscription table DBA 2016-11-04 2017-05-25
119479 Create a script that'll allow easier swatting of queries that're overloading labsdb DBA 2017-01-02 2015-11-24
134476 Decommission old coredb machines (<=db1050) DBA 2017-01-02 2017-07-24
119626 Eliminate SPOF at the main database infrastructure DBA 2017-01-02 2017-05-23
141252 icinga hp raid check timeout on busy ms-be and db machines DBA 2017-01-02 2017-08-10
132838 Missing / Dropped databases? DBA 2017-01-02 2017-07-06
140302 plwiki API request is excessively slow when including a badrevid DBA 2017-01-02 2016-08-24
127570 Rename be_x_oldwiki database to be_taraskwiki DBA 2017-01-02 2017-06-23
105395 UserMerge using SELECT img_name FROM image WHERE img_user = ... but img_user doesn't have an index DBA 2017-02-20 2015-07-09
131146 Analyze memory usage on db1011 DBA 2017-03-26 2016-03-29
121306 Error: 2013 Lost connection to MySQL server during query on IndexPager::buildQueryInfo (LogPager) DBA 2017-04-02 2017-05-08
159430 convert dbstore1001 to InnoDB compressed by importing db shards to it DBA 2017-04-14 2017-04-05
138517 mysqldump is timing out preventing all tables from being included in the dump DBA 2017-04-20 2017-04-20
164167 change_tag table needs redesign DBA 2017-05-01 2017-06-07
163143 dbtree: don't return 200 on error pages DBA 2017-05-11 2017-05-11
156465 For switchovers: A way to check if slaves are up to date DBA 2017-05-15 2017-01-27
165358 Set up and package wmfmariadbpy helper scripts so they can easily be deployed to all database server and client hosts DBA 2017-05-16 2017-05-15
50930 Database replication problems - production and labs (tracking) DBA 2017-05-25 2017-03-23
133333 Audit MySQL configurations DBA 2017-06-01 2016-11-10
115081 Pages where page.page_latest = 0 on the English Wikipedia DBA 2017-06-12 2015-12-05
59617 Make watchlist table available as curated foo_p.watchlist_count on labsdb DBA 2017-06-14 2017-03-03
154490 Delete prefstats tables DBA 2017-06-22 2017-01-03
168584 Labsdb* servers need to be rebooted DBA 2017-06-22 2017-07-14
168661 Apply schema change to add 3D filetype for STL files DBA 2017-06-23 2017-07-19
127363 mysqlbinlog doesn't recognize the ssl-* options in [client] DBA 2017-07-04 2017-01-06
114117 Drop externallinks.el_from_namespace on wmf databases DBA 2017-07-12 2017-04-28
170666 Refactor prometheus-mysqld-exporter to support multi-instance hosts DBA 2017-07-15 2017-07-25
134809 Apache <=> mariadb SSL/TLS for cross-datacenter writes DBA 2017-07-16 2017-07-25
170952 Inconsistent default charset for analytics slaves DBA 2017-07-19 2017-07-19
171643 Clean up skin properties DBA 2017-07-26 2017-07-31
171570 Rename database labswiki to wikitech DBA 2017-07-26 2017-07-25
172015 wikibase-addUsagesForPage doesn't batch properly DBA 2017-07-29 2017-07-31
162558 Use memcached (or something similar) to keep the latest chd_seen state, only flush to table every once in a while DBA 2017-08-03 2017-05-05
172497 Change pt-heartbeat model to not use super-user, avoid SPOF and switch automatically to the real master without puppet dependency DBA 2017-08-05 2017-08-07
164382 Evaluate the need for FORCE INDEX (ls_field_val) [now IGNORE INDEX (ls_log_id)], delete the index hint if not needed anymore DBA 2017-08-05 2017-05-08
146570 Give user_properties a primary key DBA 2017-08-05 2017-08-05
146391 eeden ethernet outage Network Operations 2017-01-02 2017-06-27
122406 Consider renumbering Labs to separate address spaces Network Operations 2017-04-04 2017-07-17
170131 Recurring varnish-be fetch failures in codfw Network Operations 2017-07-11 2017-07-12
130617 Collect metrics on pool counter usage Other Operations 2017-01-02 2016-11-03
134551 Create functional cluster checks for all services (and have them page!) Other Operations 2017-01-02 2017-07-12
152439 cronspam from labtestservices2001 /etc/dns-floating-ip-updater.py > /dev/null Other Operations 2017-01-02 2016-12-05
135318 Document how to handle 'inconsistent state within the internal storage backends' issues Other Operations 2017-01-02 2016-07-31
151045 Extending Yubico 2FA for production use (meta bug) Other Operations 2017-01-02 2016-11-18
146914 grain-ensure erroneous mismatch with (bool)True vs (str)true Other Operations 2017-01-02 2017-04-21
106937 Monitor [[Special:ListFiles]] for non 200 HTTP statuses in thumbnails Other Operations 2017-01-02 2017-08-03
136311 Monitor the BMC's event log for hardware errors Other Operations 2017-01-02 2016-05-26
129224 on labcontrol1001, /var/cache/salt has too many files! Other Operations 2017-01-02 2016-03-08
150672 Provide a /parsoid directory on releases.wikimedia.org Other Operations 2017-01-02 2017-05-20
119846 Redirect revisions from svn.wikimedia.org to https://phabricator.wikimedia.org/rSVN Other Operations 2017-01-02 2015-11-30
144539 Remove /srv/deployment/wdqs/wdqs/rules.log symlink Other Operations 2017-01-02 2016-10-11
141897 Review new service 'pre-deployment to production' checklist Other Operations 2017-01-02 2017-07-12
150771 Secondary production Jenkins for CI Other Operations 2017-01-02 2017-05-16
122210 Security audit for tftp on Carbon Other Operations 2017-01-02 2016-03-29
122825 Service Ownership and Maintenance Other Operations 2017-01-02 2017-08-08
101141 udp rcvbuferrors and inerrors on graphite1001 Other Operations 2017-01-02 2017-07-10
97909 Upgrade jobrunners to redis 2.8 Other Operations 2017-01-03 2016-02-13
154915 Get rid of "import realm.pp" in manifests/site.pp Other Operations 2017-01-10 2017-07-06
133913 Completely port l10nupdate to scap Other Operations 2017-01-17 2017-03-06
124413 confctl should provide tags information after writing data Other Operations 2017-02-11 2016-01-22
146664 Limit resources used by ORES Other Operations 2017-02-15 2017-05-01
120165 Implement role based hiera lookups for labs Other Operations 2017-02-27 2017-03-29
159242 Segmentation fault creating thumbnail Other Operations 2017-03-01 2017-02-28
160158 Make disabled accounts visible in the corp mirror LDAP replica Other Operations 2017-03-11 2017-03-28
150460 Configure maps cluster to send statsd metrics to the statsd endpoint in the same datacenter Other Operations 2017-03-16 2016-11-10
141038 implement icinga paging for non-ops teams Other Operations 2017-04-01 2016-07-21
161920 logrotate for ruthenium Other Operations 2017-04-01 2017-05-10
161918 videoscalers (mw1168, mw1169) - high load / overheating Other Operations 2017-04-01 2017-04-01
162029 Migrate all jessie hosts to Linux 4.9 Other Operations 2017-04-04 2017-04-13
151049 Run systematic availability tests Other Operations 2017-04-12 2016-11-18
163288 Decide on /var/lib vs /home as locations of homedir for l10nupdate Other Operations 2017-04-20 2017-04-19
76306 Set warning thresholds for average cluster utilization Other Operations 2017-04-21 2016-12-16
141756 audit / test / upgrade hp smartarray P840 firmware Other Operations 2017-04-22 2017-06-06
97524 ocg alarm ocg_job_status_queue 'flapping' Other Operations 2017-04-28 2016-10-03
149617 Integrating MediaWiki (and other services) with dynamic configuration Other Operations 2017-05-01 2017-08-08
165173 rack/setup/install dumpsdata100[12] Other Operations 2017-05-13 2017-07-06
165520 rack and setup wtp1025-1048 Other Operations 2017-05-17 2017-07-12
151050 Proper documentation for Yubico 2FA for production use Other Operations 2017-05-20 2016-11-18
165781 rack/setup/install labcontrol100[34] Other Operations 2017-05-20 2017-07-12
165784 rack/setup/install labmon1002 Other Operations 2017-05-20 2017-08-03
133656 Have a paging check for Nova API accessible Other Operations 2017-05-22 2016-12-21
166038 Sync internal nutcracker package with Debian package Other Operations 2017-05-23 2017-05-29
166322 spam from phabricator in labs Other Operations 2017-05-26 2017-05-26
135124 Deploy etcddump (or another etcd dump & load tool) to production Other Operations 2017-05-28 2016-05-12
116580 monitor postgresql replication status Other Operations 2017-05-31 2015-10-26
153416 docker-engine pulled into our repositories only keeps the latest version Other Operations 2017-06-06 2017-01-04
135338 On Trusty and Jessie PHP yields: PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/20-xhprof.ini on line 2 Other Operations 2017-06-06 2017-07-01
164290 Set up external DNS record for wikitech-static Other Operations 2017-06-07 2017-06-06
156136 Increase swift replication factor for accounts Other Operations 2017-06-10 2017-01-24
165348 Check long-running screen/tmux sessions Other Operations 2017-06-11 2017-07-10
161528 incident 20170323-wikibase did not trigger Icinga paging Other Operations 2017-06-14 2017-03-27
167820 rack/setup/install labweb100[12].wikimedia.org Other Operations 2017-06-14 2017-07-14
167992 rack/setup/install new kafka nodes kafka-jumbo100[1-6] Other Operations 2017-06-16 2017-08-09
152562 Port fundraising stats off Ganglia Other Operations 2017-06-19 2017-06-27
160101 Upgrade php5-json .deb to at least 1.3.8 Other Operations 2017-06-20 2017-03-09
168619 Degraded RAID on lvs3001 Other Operations 2017-06-23 2017-07-19
113104 Set up a service IP for logstash Other Operations 2017-06-24 2017-06-23
140442 reinstall rdb100[56] with RAID Other Operations 2017-06-27 2017-04-18
169035 bast3002 sdb broken Other Operations 2017-06-29 2017-07-03
160941 Improve SSH access information in onboarding documentation Other Operations 2017-06-30 2017-03-30
115757 document debian packaging guidelines Other Operations 2017-07-02 2016-05-23
119401 Untangle labs/production roles from labs/instance roles Other Operations 2017-07-03 2015-11-23
167412 host-vmem.erb is doing operations that make no sense Other Operations 2017-07-04 2017-06-17
169658 Improve database backups' coverage, monitoring and data recovery time (part 1) (tracking) Other Operations 2017-07-05 2017-07-04
140813 Protect sensitive user-related information with a UserData / auth / session service Other Operations 2017-07-06 2017-06-08
121610 system users with UIDs > 500 Other Operations 2017-07-09 2015-12-16
95054 Move ircecho config file to be YAML Other Operations 2017-07-10 2016-07-15
170120 Standardize on the "default" pod setup Other Operations 2017-07-11 2017-07-10
104671 Rename 'restricted' group? Other Operations 2017-07-12 2017-07-11
147872 Rename rhodium to puppetmaster1003 Other Operations 2017-07-12 2016-10-11
126158 [RFC] Alert about *when* partitions will run out of space, not a percentage/absolute number Other Operations 2017-07-12 2016-05-02
170740 PuppetDB misbehaving on 2017-07-15 Other Operations 2017-07-16 2017-07-19
129841 Many minions fail to connect to salt master since 10:39 Other Operations 2017-07-17 2016-03-14
158429 Switch to predictable network interface names? Other Operations 2017-07-18 2017-06-07
147366 Setup automated topk wide row reporting Other Operations 2017-07-19 2017-07-18
133476 Proposal: Centralize OTRS login methodology Other Operations 2017-07-20 2016-05-21
86081 Complete the use of HHVM over Zend PHP on the Wikimedia cluster Other Operations 2017-07-21 2017-08-01
125442 es2009 degraded RAID Other Operations 2017-07-22 2016-04-22
165618 Audit / document reasons for not enabling HT? Other Operations 2017-07-24 2017-05-17
171623 Split up labstore external shelf storage available in codfw between labstore2001 and 2 Other Operations 2017-07-26 2017-07-25
120943 Wikimania 2017 site does not automatically redirect to mobile site, when opening from a mobile device Other Operations 2017-07-26 2017-07-25
171188 Move the main WMCS puppetmaster into the Labs realm Other Operations 2017-08-03 2017-07-24
162123 Running swiftrepl is not puppetized Other Operations 2017-08-04 2017-07-06
116627 Include 5xx numbers in fluorine fatalmonitor Other Operations 2017-08-08 2015-10-28
159536 Puppet constantly trying to stop the already stopped puppetmaster process on Trusty Other Operations 2017-08-10 2017-03-03
159045 Update Puppet repo code that uses maniphest.update and maniphest.createtask conduit api Software Development 2017-05-12 2017-02-25
148494 Add shell scripts CI validations Software Development 2017-05-28 2016-12-15
159127 Cumin: fine tuning configuration Software Development 2017-06-09 2017-02-27
150560 More verbose messages from service-checker-swagger Software Development 2017-06-23 2017-05-23
169304 Cumin masters: simplify usage in case of emergency Software Development 2017-07-01 2017-06-30
152950 E901 SyntaxError: invalid syntax is wrongly raised on using python's abc by jenkins python CI linter Software Development 2017-07-22 2017-03-27
109331 Deleted files sometimes remain visible to non-privileged users if permanently linked Traffic 2017-01-02 2017-07-27
102178 Fix RESTBase support for wikitech.wikimedia.org Traffic 2017-01-02 2017-07-11
137979 Support brotli compression Traffic 2017-01-02 2017-08-08
154801 Investigate varnishd child crashes when multiple nodes get depooled/pooled concurrently Traffic 2017-01-07 2017-02-07
141266 letsencrypt puppetization: add parallel rsa+ecdsa cert support Traffic 2017-01-23 2016-09-30
156256 Select or Acquire Address Space for Asia Cache DC Traffic 2017-01-26 2017-04-11
157786 Unhandled error stopping pybal: 'RunCommandMonitoringProtocol' object has no attribute 'checkCall' Traffic 2017-02-11 2017-02-10
75944 Monitor Varnish caches on beta cluster have two varnishd process running Traffic 2017-02-24 2016-09-30
96853 Evaluate Apache Traffic Server Traffic 2017-04-15 2017-06-21
152882 Many misc wikis lack mobile domains Traffic 2017-04-15 2017-07-25
128559 store.wikimedia.org HTTPS issues Traffic 2017-04-15 2017-07-11
109776 Tilerator should purge Varnish cache Traffic 2017-04-30 2017-03-08
164259 Add VSL error counters to Varnishkafka stats Traffic 2017-05-03 2017-08-04
164609 Merge cache_misc into cache_text functionally Traffic 2017-05-06 2017-06-27
144508 Point wikipedia.in to 205.147.101.160 instead of URL forward Traffic 2017-05-19 2016-10-04
165765 Refactor pybal/LVS config for shared failover Traffic 2017-05-20 2017-06-01
119372 Pybal IdleConnectionMonitor with TCP KeepAlive shows random fails if more than 100 servers are involved. Traffic 2017-05-24 2017-03-30
106517 upload.wikimedia.org returns HTTP status code 503 for truncated urls, not 404 Traffic 2017-05-24 2016-09-30
154702 Fix broken referer categorization for visits from Safari browsers Traffic 2017-05-25 2017-01-20
112316 Configure varnish to use "Unconfigured domain" page for 404 Not Served (instead of generic error) Traffic 2017-05-27 2016-09-30
162684 Network hardware configuration for Asia Cache DC Traffic 2017-06-05 2017-04-11
120121 Improve Varnish XFF processing for trusted proxies Traffic 2017-06-09 2016-10-04
164868 SSL error for https://wikispecies.org/ Traffic 2017-06-10 2017-06-20
120486 add a https-only option to dynamicproxy Traffic 2017-06-18 2016-10-04
168529 Upgrade to Varnish 5 Traffic 2017-06-22 2017-08-08
136944 Set up LVS connection sync Traffic 2017-07-02 2016-09-30
96852 Define 3-host infra cluster for traffic pops Traffic 2017-07-03 2017-03-21
79730 Add pybal check to ensure service IP is bound Traffic 2017-07-05 2017-03-30
169765 pybal should automatically reconnect to etcd Traffic 2017-07-06 2017-07-18
166782 wikimediafoundation.org's language selector is confusing to most visitors who don't have accounts there Traffic 2017-07-07 2017-07-21
112765 Phabricator needs to expose notification daemon (websocket) Traffic 2017-07-17 2016-11-15
141480 mixed-content issues on planet.wikimedia.org Traffic 2017-08-06 2016-10-04
165764 Fully-redundant LVS clusters using Pybal per-service MED feature Traffic 2017-08-08 2017-05-22
137747 Parametrization of VCL is inconsistent Traffic 2017-08-08 2016-09-30
148976 Strongswan Icinga check: do not report issues about depooled hosts Traffic 2017-08-08 2016-10-24