TechnicalOperations Unpointed Tasks

Help.

122406 Consider renumbering Labs to separate address spaces Network Operations open 2017-04-04 2017-07-17
146391 eeden ethernet outage Network Operations open 2017-01-02 2017-06-27
170131 Recurring varnish-be fetch failures in codfw Network Operations open 2017-07-11 2017-07-12
80253 Fix esams management network Network Operations resolved 2017-01-02 2015-09-08
83898 JSNMP flood of errors across multiple switches Network Operations resolved 2017-06-24 2017-05-23
89294 Establish IPsec tunnel between codfw and eqiad pfw Network Operations resolved 2017-04-27 2015-07-09
92548 Network congestion between DTAG & eqiad Network Operations resolved 2017-02-08 2015-03-17
104458 rack/setup new eqiad lvs machines Network Operations resolved 2017-01-02 2016-11-08
116766 attach mr1-codfw to new out of band mgmt link Network Operations resolved 2017-05-27 2015-10-27
118256 Set up cr2-esams Network Operations resolved 2017-01-02 2016-05-30
118259 Figure out the source of QSFP+ errors with DAC + MX480 Network Operations resolved 2017-07-12 2015-11-25
127295 mr1-ulsfo booted from backup JunOS Network Operations resolved 2017-07-17 2016-02-18
133852 analytics hosts frequently tripping 'port utilization threshold' librenms alerts Network Operations resolved 2017-08-05 2017-05-23
134932 cr2-codfw LUCHIP/trinity_pio error messages Network Operations resolved 2017-04-29 2016-06-01
143248 configure port for frdb1001 Network Operations resolved 2017-02-06 2016-09-08
143915 Fix static IP fallbacks to Pybal LVS routes Network Operations resolved 2017-08-01 2016-10-26
145270 Telkom/8ta (South Africa) users cannot connect to wikimedia sites Network Operations resolved 2017-07-14 2016-09-19
149829 Move labsdb1008 to production, rename it back to db1095, use it as a temporary sanitarium Network Operations resolved 2017-01-02 2016-11-15
150400 Spare pool servers switch configuration Network Operations resolved 2017-05-02 2016-11-10
155875 asw-c2-eqiad reboots & fdb_mac_entry_mc_set() issues Network Operations resolved 2017-08-05 2017-02-17
156957 asw-d-codfw public1-vlan addition review (blocks gerrit2001) Network Operations resolved 2017-07-30 2017-02-02
164594 codfw: ganeti2007-ganeti2008 switch port configuration Network Operations resolved 2017-05-13 2017-05-08
171714 "MySQL server has gone away" from librenms logs Network Operations resolved 2017-07-27 2017-08-02
75944 Monitor Varnish caches on beta cluster have two varnishd process running Traffic open 2017-02-24 2016-09-30
79730 Add pybal check to ensure service IP is bound Traffic open 2017-07-05 2017-03-30
96852 Define 3-host infra cluster for traffic pops Traffic open 2017-07-03 2017-03-21
96853 Evaluate Apache Traffic Server Traffic open 2017-04-15 2017-06-21
102178 Fix RESTBase support for wikitech.wikimedia.org Traffic open 2017-01-02 2017-07-11
106517 upload.wikimedia.org returns HTTP status code 503 for truncated urls, not 404 Traffic open 2017-05-24 2016-09-30
109331 Deleted files sometimes remain visible to non-privileged users if permanently linked Traffic open 2017-01-02 2017-07-27
109776 Tilerator should purge Varnish cache Traffic open 2017-04-30 2017-03-08
112316 Configure varnish to use "Unconfigured domain" page for 404 Not Served (instead of generic error) Traffic open 2017-05-27 2016-09-30
112765 Phabricator needs to expose notification daemon (websocket) Traffic open 2017-07-17 2016-11-15
119372 Pybal IdleConnectionMonitor with TCP KeepAlive shows random fails if more than 100 servers are involved. Traffic open 2017-05-24 2017-03-30
120121 Improve Varnish XFF processing for trusted proxies Traffic open 2017-06-09 2016-10-04
120486 add a https-only option to dynamicproxy Traffic open 2017-06-18 2016-10-04
128559 store.wikimedia.org HTTPS issues Traffic open 2017-04-15 2017-07-11
136944 Set up LVS connection sync Traffic open 2017-07-02 2016-09-30
137747 Parametrization of VCL is inconsistent Traffic open 2017-08-08 2016-09-30
137979 Support brotli compression Traffic open 2017-01-02 2017-08-08
141266 letsencrypt puppetization: add parallel rsa+ecdsa cert support Traffic open 2017-01-23 2016-09-30
141480 mixed-content issues on planet.wikimedia.org Traffic open 2017-08-06 2016-10-04
144508 Point wikipedia.in to 205.147.101.160 instead of URL forward Traffic open 2017-05-19 2016-10-04
148976 Strongswan Icinga check: do not report issues about depooled hosts Traffic open 2017-08-08 2016-10-24
152882 Many misc wikis lack mobile domains Traffic open 2017-04-15 2017-07-25
154702 Fix broken referer categorization for visits from Safari browsers Traffic open 2017-05-25 2017-01-20
154801 Investigate varnishd child crashes when multiple nodes get depooled/pooled concurrently Traffic open 2017-01-07 2017-02-07
156256 Select or Acquire Address Space for Asia Cache DC Traffic open 2017-01-26 2017-04-11
157786 Unhandled error stopping pybal: 'RunCommandMonitoringProtocol' object has no attribute 'checkCall' Traffic open 2017-02-11 2017-02-10
162684 Network hardware configuration for Asia Cache DC Traffic open 2017-06-05 2017-04-11
164259 Add VSL error counters to Varnishkafka stats Traffic open 2017-05-03 2017-08-04
164609 Merge cache_misc into cache_text functionally Traffic open 2017-05-06 2017-06-27
164868 SSL error for https://wikispecies.org/ Traffic open 2017-06-10 2017-06-20
165764 Fully-redundant LVS clusters using Pybal per-service MED feature Traffic open 2017-08-08 2017-05-22
165765 Refactor pybal/LVS config for shared failover Traffic open 2017-05-20 2017-06-01
166782 wikimediafoundation.org's language selector is confusing to most visitors who don't have accounts there Traffic open 2017-07-07 2017-07-21
168529 Upgrade to Varnish 5 Traffic open 2017-06-22 2017-08-08
169765 pybal should automatically reconnect to etcd Traffic open 2017-07-06 2017-07-18
20496 Non-secure logos on secure pages Traffic resolved 2017-01-02 2016-08-05
63443 Enable gzip transfer encoding for png and ico files in /static Traffic resolved 2017-01-02 2016-05-04
82747 pybal health checks are ipv4 even for ipv6 vips Traffic resolved 2017-08-08 2017-07-18
86664 HTTPS performance & UA adoption metrics Traffic resolved 2017-04-26 2015-08-27
89688 Varnish GeoIP is broken for HTTPS+IPv6 traffic Traffic resolved 2017-08-11 2016-08-18
93927 Make OCSP Stapling support more generic and robust Traffic resolved 2017-07-11 2016-11-15
96375 Upgrade codfw,ulsfo,esams LVS to jessie Traffic resolved 2017-01-02 2015-09-24
101435 SPDY traffic isn't gzipped for Firefox Traffic resolved 2017-08-05 2015-06-08
101892 Ensure {text-domain}/w/load.php requests do not bypass cache for session cookies Traffic resolved 2017-01-02 2015-06-28
102394 Implement pybal pool state monitoring and alerting via icinga Traffic resolved 2017-04-26 2016-02-09
102815 Decom www.$lang hostnames/redirects Traffic resolved 2017-07-01 2015-09-14
103773 check if services behind misc-web enforce http->https redirect or not Traffic resolved 2017-07-26 2015-06-28
104076 Puppet failing on deployment-prep caches Traffic resolved 2017-02-19 2015-09-12
105794 Insecure POST traffic Traffic resolved 2017-01-27 2016-09-23
107060 DNS request for wikimedia.org Traffic resolved 2017-04-27 2015-08-07
107086 Provide an API listing at /api/ Traffic resolved 2017-01-02 2016-02-29
107430 Decom bits.wikimedia.org hostname Traffic resolved 2017-01-31 2017-04-06
108649 Set up "w.wiki" domain for usage with UrlShortener Traffic resolved 2017-06-28 2015-12-11
109738 ocsp updater: validate the signature expiry lifetime Traffic resolved 2017-05-28 2016-10-03
109740 ocsp updater: re-enable automatic updates Traffic resolved 2017-01-02 2015-08-27
111045 zerofetcher in production is getting throttled for API logins Traffic resolved 2017-06-11 2015-11-30
112195 Beta giving Error: 403, Insecure POST Forbidden Traffic resolved 2017-05-27 2015-09-22
113007 [Research] Investigate 30% load.php reqs increase since 2015-07-30 Traffic resolved 2017-03-29 2016-08-19
113154 LVS HTTPS IPv6 on mobile-lb.eqiad alert occasionally flapping Traffic resolved 2017-01-02 2015-10-05
114995 [Bug] Varnish cache invalidation not working for m.wikidata.org Traffic resolved 2017-07-29 2016-05-04
118787 releases.wikimedia.org should be https only and have hsts set Traffic resolved 2017-07-23 2017-06-07
118831 Upgrade the production etcd cluster to 2.2 Traffic resolved 2017-01-02 2016-01-13
122455 Varnish leaks memory Traffic resolved 2017-02-21 2016-05-31
123431 Investigate if login.m.wikimedia.org needs to stay around Traffic resolved 2017-04-13 2016-03-28
124356 Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) Traffic resolved 2017-01-02 2017-07-27
124835 Automate and/or better-document varnish ban procedure for operations staff, so it can be accomplished with more speed and confidence in outage conditions Traffic resolved 2017-02-03 2016-01-27
126053 Send X-Carrier + X-Carrier-Meta headers on all responses Traffic resolved 2017-01-28 2016-02-29
126062 cp30[34]x hw/firmware/BMC issues Traffic resolved 2017-01-02 2016-05-31
127492 Switch ulsfo to backend to codfw rather than eqiad Traffic resolved 2017-08-08 2016-03-08
128961 Images not showing up at Commons Traffic resolved 2017-01-02 2016-04-20
129270 Fix puppet on deployment-cache* hosts in beta cluster Traffic resolved 2017-01-02 2016-03-25
131501 Convert misc cluster to Varnish 4 Traffic resolved 2017-07-04 2016-07-27
132346 cronspam from cpXXXX hosts related to varnishkafka non existent processes Traffic resolved 2017-01-02 2016-04-27
132430 varnishstatsd crashes with ValueError in vsl_callback without being restarted by systemd Traffic resolved 2017-01-02 2016-05-11
132462 HTTPS redirects for parsoid-tests.wikimedia.org Traffic resolved 2017-07-03 2016-04-13
132464 HTTPS redirects for transparency.wikimedia.org Traffic resolved 2017-07-07 2016-04-13
132474 varnishmedia: repeated calls to flush_stats() Traffic resolved 2017-03-11 2016-06-17
132685 Preload STS for wikimedia.org Traffic resolved 2017-06-09 2016-06-17
133217 Fix apache-2.4 + DHE ciphersuites issue Traffic resolved 2017-07-31 2016-04-26
133490 Wikidata Query Service REST endpoint returns truncated results Traffic resolved 2017-06-26 2016-05-09
133915 lvs1012 eth1 NIC Link flapping Traffic resolved 2017-01-02 2016-04-28
133936 Beta cluster varnish sets overly broad domain on GeoIP cookie Traffic resolved 2017-07-12 2016-08-18
134346 varnish text on beta is unreachable / stuck Traffic resolved 2017-01-02 2016-06-30
135515 graphite.wikimedia.org 503s on some css/js resources Traffic resolved 2017-07-20 2016-05-18
138682 https://ores.wikimedia.org redirects me to HTTP when I don't include a trailing slash Traffic resolved 2017-01-02 2016-08-12
141454 piuparts fail with WARN: Broken symlinks: /etc/systemd/system... Traffic resolved 2017-06-26 2016-11-18
142848 Stop using persistent storage in our backend varnish layers. Traffic resolved 2017-01-02 2016-10-03
142944 Performance and caching considerations for article placeholders accesses Traffic resolved 2017-01-02 2017-01-31
144293 adjust ssl certificate montioring to differentiate between standard and LE certificates. Traffic resolved 2017-02-23 2016-11-16
146451 repeated 503 errors for 90 minutes now Traffic resolved 2017-07-08 2016-11-26
147784 Mediawiki thumbnail requests for 0px should result in http 400 not 500 Traffic resolved 2017-01-02 2016-12-15
148780 mobile-safari has very few internally-referred pageviews Traffic resolved 2017-04-02 2017-02-15
149843 reimage cp4016 and cp1055 Traffic resolved 2017-01-02 2016-11-07
150879 Cannot connect to cp3039.mgmt.esams.wmnet:22 Traffic resolved 2017-07-24 2016-11-17
151561 varnishapi.py AttributeError: VSM_Close Traffic resolved 2017-03-02 2016-11-28
152690 Refreshing mobile-sections does not purge mobile-sections-lead Traffic resolved 2017-01-02 2016-12-13
155504 Fix universal link support in iOS when the OS requests the site association file from m.wikipedia.org Traffic resolved 2017-01-18 2017-01-27
155807 Monitor Certificate Transparency (CT) logs Traffic resolved 2017-06-25 2017-02-13
157353 prometheus-vhtcpd-stats cronspamming if vhtcpd is not running yet Traffic resolved 2017-02-07 2017-07-19
157398 Book collections communicate with pediapress using http: Traffic resolved 2017-02-08 2017-02-15
158638 Using wikimedia.ee mail address as Google account Traffic resolved 2017-02-22 2017-04-07
159574 LDF endpoint ordering is not stable between servers when paging Traffic resolved 2017-04-02 2017-07-14
161819 Investigate 502 errors from nginx when backend returns 302 Traffic resolved 2017-03-31 2017-04-10
164579 Investigate nginx reload behavior Traffic resolved 2017-05-06 2017-07-24
164608 Merge cache_maps into cache_upload functionally Traffic resolved 2017-05-06 2017-06-08
170192 remove eventdonations.wikimedia.org CNAME Traffic resolved 2017-07-11 2017-07-11
50930 Database replication problems - production and labs (tracking) DBA open 2017-05-25 2017-03-23
59617 Make watchlist table available as curated foo_p.watchlist_count on labsdb DBA open 2017-06-14 2017-03-03
105395 UserMerge using SELECT img_name FROM image WHERE img_user = ... but img_user doesn't have an index DBA open 2017-02-20 2015-07-09
114117 Drop externallinks.el_from_namespace on wmf databases DBA open 2017-07-12 2017-04-28
115081 Pages where page.page_latest = 0 on the English Wikipedia DBA open 2017-06-12 2015-12-05
119479 Create a script that'll allow easier swatting of queries that're overloading labsdb DBA open 2017-01-02 2015-11-24
119626 Eliminate SPOF at the main database infrastructure DBA open 2017-01-02 2017-05-23
121306 Error: 2013 Lost connection to MySQL server during query on IndexPager::buildQueryInfo (LogPager) DBA open 2017-04-02 2017-05-08
127363 mysqlbinlog doesn't recognize the ssl-* options in [client] DBA open 2017-07-04 2017-01-06
127570 Rename be_x_oldwiki database to be_taraskwiki DBA open 2017-01-02 2017-06-23
131146 Analyze memory usage on db1011 DBA open 2017-03-26 2016-03-29
132838 Missing / Dropped databases? DBA open 2017-01-02 2017-07-06
133333 Audit MySQL configurations DBA open 2017-06-01 2016-11-10
134476 Decommission old coredb machines (<=db1050) DBA open 2017-01-02 2017-07-24
134809 Apache <=> mariadb SSL/TLS for cross-datacenter writes DBA open 2017-07-16 2017-07-25
138517 mysqldump is timing out preventing all tables from being included in the dump DBA open 2017-04-20 2017-04-20
140302 plwiki API request is excessively slow when including a badrevid DBA open 2017-01-02 2016-08-24
141252 icinga hp raid check timeout on busy ms-be and db machines DBA open 2017-01-02 2017-08-10
146570 Give user_properties a primary key DBA open 2017-08-05 2017-08-05
149936 Drop flow_subscription table DBA open 2016-11-04 2017-05-25
154490 Delete prefstats tables DBA open 2017-06-22 2017-01-03
156465 For switchovers: A way to check if slaves are up to date DBA open 2017-05-15 2017-01-27
159430 convert dbstore1001 to InnoDB compressed by importing db shards to it DBA open 2017-04-14 2017-04-05
162558 Use memcached (or something similar) to keep the latest chd_seen state, only flush to table every once in a while DBA open 2017-08-03 2017-05-05
163143 dbtree: don't return 200 on error pages DBA open 2017-05-11 2017-05-11
164167 change_tag table needs redesign DBA open 2017-05-01 2017-06-07
164382 Evaluate the need for FORCE INDEX (ls_field_val) [now IGNORE INDEX (ls_log_id)], delete the index hint if not needed anymore DBA open 2017-08-05 2017-05-08
165358 Set up and package wmfmariadbpy helper scripts so they can easily be deployed to all database server and client hosts DBA open 2017-05-16 2017-05-15
168584 Labsdb* servers need to be rebooted DBA open 2017-06-22 2017-07-14
168661 Apply schema change to add 3D filetype for STL files DBA open 2017-06-23 2017-07-19
170666 Refactor prometheus-mysqld-exporter to support multi-instance hosts DBA open 2017-07-15 2017-07-25
170952 Inconsistent default charset for analytics slaves DBA open 2017-07-19 2017-07-19
171570 Rename database labswiki to wikitech DBA open 2017-07-26 2017-07-25
171643 Clean up skin properties DBA open 2017-07-26 2017-07-31
172015 wikibase-addUsagesForPage doesn't batch properly DBA open 2017-07-29 2017-07-31
172497 Change pt-heartbeat model to not use super-user, avoid SPOF and switch automatically to the real master without puppet dependency DBA open 2017-08-05 2017-08-07
12817 Filtering Special:NewPages by page size DBA resolved 2017-08-04 2016-11-10
54927 Drop database table "hashs" from Wikimedia wikis DBA resolved 2017-05-06 2017-04-27
73176 Discrepancy between enwiki_p.pagelinks on labs and production DBA resolved 2017-07-23 2015-06-26
74908 Replication behind or missing records DBA resolved 2017-07-18 2015-08-17
95871 Use External Store on Beta Cluster DBA resolved 2015-10-02 2016-08-12
98682 labswiki DB is inaccessible from tin, terbium, etc. DBA resolved 2017-01-02 2015-08-20
99459 ips_site_page is too short to store some (full) page titles DBA resolved 2017-01-02 2016-08-11
102484 Tokudb requires disabling huge pages; puppetize on labsdb* hosts DBA resolved 2017-05-19 2015-06-15
104471 Tables created on the s7 master have not been replicated to dbstore2001 and dbstore2002. Replication issue? DBA resolved 2017-01-02 2015-07-23
105768 Remove db1002-db1007 from production DBA resolved 2017-03-06 2017-02-10
105911 tools.tools-info credentials are not functioning DBA resolved 2017-01-02 2016-01-27
108648 Migrate OAuth data from mediawikiwiki to metawiki DBA resolved 2017-07-31 2016-01-06
109676 Add artificial primary key to flow_wiki_ref and flow_ext_ref DBA resolved 2015-10-02 2016-03-01
111992 Physical location SPOF because of database server distribution on a single rack (D1) DBA resolved 2017-08-06 2016-06-16
112041 New database for designate pool manager DBA resolved 2017-03-24 2015-09-10
112479 Check, test and tune pool-of-connections and max_connections configuration DBA resolved 2017-02-06 2015-10-29
114068 Clean up search user preferences in user_properties table on Wikimedia wikis DBA resolved 2017-01-02 2015-09-29
118816 mysql privs: restrict access to racktables to krypton DBA resolved 2017-05-22 2016-01-29
119056 External Storage on codfw (es2005-2010) is consuming 100-90GB of disk space per server and per month and it has 370GB available DBA resolved 2017-08-07 2016-03-10
119619 Decide storage backend for performance schema monitoring stats DBA resolved 2017-06-02 2017-02-14
120122 Perform a rolling restart of all MySQL slaves (masters too for those services with low traffic) DBA resolved 2017-05-20 2016-04-22
122111 ref_target should be NOT NULL DBA resolved 2015-12-23 2016-02-25
123731 Migrate labsdb1005/1006/1007 to jessie DBA resolved 2017-05-05 2017-03-31
124704 puppetize mysql m5-master access on parsoid-test server DBA resolved 2017-05-31 2016-01-27
124705 New database for access from stat1003 (project_illustration) DBA resolved 2017-01-02 2016-01-25
125215 Prepare db1018 and s2-slaves for s2 master failover DBA resolved 2017-08-07 2016-02-10
125386 Clarify how mysql dumps will be architectured during codfw failover DBA resolved 2017-01-08 2016-02-08
127107 Copy user database to c1 DBA resolved 2017-08-01 2016-02-17
127811 MariaDB package improvements DBA resolved 2017-05-15 2016-11-17
130702 Several es20XX servers keep crashing (es2017, es2019, es2015, es2014) since 23 March DBA resolved 2017-04-24 2017-06-06
131753 Semi-synchronous replication status in all MySQL production clusters DBA resolved 2017-06-30 2017-03-23
132279 db1053 transient replica lag during dumping DBA resolved 2017-04-18 2016-08-02
133205 Switchback to eqiad DBA resolved 2017-08-03 2016-04-21
133309 db1041 regular mysql stalls creates connection failures DBA resolved 2017-04-18 2016-04-22
133322 u3532__ (=marcmiquel) table using 64G on labsdb1001 and 108 GB on labsdb1003 DBA resolved 2017-06-04 2017-03-28
133584 Reduce stress caused by IABot on the DB DBA resolved 2017-01-02 2016-08-05
135027 MASTER_POS_WAIT() alternative that works cross-DC DBA resolved 2017-01-02 2017-04-03
135471 Hundreds of concurrent SELECT MASTER_POS_WAIT('db1049-bin.002927', 354219766, 10) without debugging information (no user, not function) on s5 API node DBA resolved 2017-04-30 2016-05-26
136618 Wrong page title in labs database replica enwiki page table DBA resolved 2017-01-02 2017-03-03
138460 Upgrade m3 (phabricator) db servers DBA resolved 2017-07-13 2016-08-25
139346 All s4 (commons) slaves failed replication DBA resolved 2017-01-09 2016-07-08
140537 Query for Special:Contributions?contribs=newbie&dir=prev is slow on enwiki DBA resolved 2017-05-08 2016-07-19
142223 Enable access to Wikipedia Tulu (tcywiki) on labs replicas DBA resolved 2017-01-02 2016-12-17
143934 s2 replag currently 8 hours DBA resolved 2017-07-25 2016-08-26
145630 Cannot delete two pages with large histories even having the appropriate permissions to do so DBA resolved 2017-07-17 2016-09-14
146018 Investigate db1061 crash DBA resolved 2017-01-02 2017-04-05
146111 hidenondamaging=1 query is extremely slow on enwiki DBA resolved 2017-01-02 2016-10-11
146529 5-10 times more rows are currently read from the s5 master than from all other masters together DBA resolved 2017-03-30 2016-09-28
147052 Provision with data the new labsdb servers and provide replica service with at least 1 shard from a sanitized copy from production DBA resolved 2017-07-08 2016-12-21
147172 db1055: degraded array DBA resolved 2017-06-25 2016-10-06
147774 Physically move db1053 to a different rack DBA resolved 2017-04-11 2016-10-13
148790 apply events killing long running queries to db1070; any other production server DBA resolved 2017-04-05 2016-10-28
150960 Initial data tests for db1095 (temporary db1069 - sanitarium replacement) DBA resolved 2017-06-30 2016-12-07
150974 db2042 disk predictive failure DBA resolved 2017-07-22 2016-12-07
151076 labsdbadmin needs to be able to drop users and update passwords DBA resolved 2017-07-07 2016-11-18
151272 db1092 crash DBA resolved 2017-01-26 2016-11-28
152761 db1028 increased lag after extensions/CentralAuth/maintenance/populateLocalAndGlobalIds.php DBA resolved 2017-01-30 2016-12-15
153300 Remove partitions from metawiki.pagelinks in s7 DBA resolved 2017-01-24 2017-04-27
153739 Defragment db1015 DBA resolved 2017-07-11 2017-01-26
153826 Defragment db1044 DBA resolved 2017-06-12 2017-01-18
154097 Remove partitions from enwiktionary.templatelinks in s2 DBA resolved 2017-01-24 2017-03-28
156006 Move db1052 to row B3 DBA resolved 2017-06-08 2017-01-24
156226 Reimage and clone db1072 DBA resolved 2017-01-26 2017-02-06
156758 Drop m3 from dbstore servers DBA resolved 2017-02-01 2017-08-10
156905 Phabricator master and slave crashed DBA resolved 2017-02-02 2017-02-14
159414 Rampant differences in indexes and PK on s6 (frwiki, jawiki, ruwiki) for revision table DBA resolved 2017-08-10 2017-03-14
160435 db1057 does not react to powercycle/powerdown/powerup commands DBA resolved 2017-03-16 2017-04-04
160713 Replicate babel db table on Labs DBA resolved 2017-03-17 2017-07-07
160731 db1048 BBU Faulty - slave lagging DBA resolved 2017-03-18 2017-05-31
160984 Reduce max execution time of interactive queries or a better detection and killing of bad query patterns DBA resolved 2017-04-18 2017-08-07
161524 Implements ContentHandler abstraction for ProofreadPage Index: pages DBA resolved 2017-06-03 2017-08-02
162681 Network maintenance on row D (databases) DBA resolved 2017-04-12 2017-04-28
166141 db1046 BBU looks faulty DBA resolved 2017-05-24 2017-07-06
166546 Point labsdb1001 and labsdb1003 to db1095 and db1102 DBA resolved 2017-05-31 2017-08-09
167031 Global rename of Idh0854 → Garam: supervision needed DBA resolved 2017-06-06 2017-07-10
168021 setup dewiki and wikidatawiki on the labsdb1009, 1010 and 1011 DBA resolved 2017-06-16 2017-06-20
170351 db2019 has performance issues, replace disk or switchover s4 master elsewhere DBA resolved 2017-07-12 2017-08-04
170941 Global rename of user Moros DBA resolved 2017-07-19 2017-07-19
148494 Add shell scripts CI validations Software Development open 2017-05-28 2016-12-15
150560 More verbose messages from service-checker-swagger Software Development open 2017-06-23 2017-05-23
152950 E901 SyntaxError: invalid syntax is wrongly raised on using python's abc by jenkins python CI linter Software Development open 2017-07-22 2017-03-27
159045 Update Puppet repo code that uses maniphest.update and maniphest.createtask conduit api Software Development open 2017-05-12 2017-02-25
159127 Cumin: fine tuning configuration Software Development open 2017-06-09 2017-02-27
169304 Cumin masters: simplify usage in case of emergency Software Development open 2017-07-01 2017-06-30
149913 Icinga raid_handler: add option to handle frack instances Software Development resolved 2017-03-13 2017-02-24
158747 Cumin: better error message if no config file is available Software Development resolved 2017-06-25 2017-07-03
161887 Cumin: CLI, allow to just query without specifying a command Software Development resolved 2017-05-30 2017-03-31
164403 Switchdc t09 start maintenance: clear systemctl failed state Software Development resolved 2017-05-19 2017-05-08
164838 Cumin: allow to specify a timeout per command Software Development resolved 2017-05-10 2017-07-03
84958 eqiad: (3) servers for logstash service Hardware Requests resolved 2017-06-13 2016-07-29
90904 deploy eventlog1001 Hardware Requests resolved 2017-01-02 2015-04-07
94846 Buy fiber patches Hardware Requests resolved 2017-08-11 2015-11-03
95153 Decom/repurpose rbf* hosts Hardware Requests resolved 2017-03-08 2015-09-16
100442 Hadoop worker node procurement - 2015 Hardware Requests resolved 2017-07-09 2016-01-27
111777 Refresh Parser cache servers pc1001-pc1003 Hardware Requests resolved 2017-01-02 2015-12-18
112163 eqiad: (1) hardware request for ElasticSearch replication to Labs - 4 weeks use Hardware Requests resolved 2017-08-01 2015-10-16
118588 Get Ops bare metal test server Hardware Requests resolved 2017-04-30 2015-12-03
119598 eqiad: (2) servers request for ORES Hardware Requests resolved 2017-01-02 2016-08-05
119698 spare swift disks order Hardware Requests resolved 2017-05-07 2015-12-22
120842 EQIAD/CODFW: 2 hardware access request for monitoring Hardware Requests resolved 2017-05-28 2015-12-15
121007 Decommission and remove from racks out of warranty spares Hardware Requests resolved 2017-06-17 2015-12-17
122100 wipe disks and add nembus back to server spares Hardware Requests resolved 2017-04-19 2017-04-13
126453 codfw: (2) servers for redis jobrunners Hardware Requests resolved 2017-04-01 2016-03-12
130752 reclaim restbase1001-1006 to spares Hardware Requests resolved 2017-01-02 2016-05-16
139758 Decommission elastic1001-1016 Hardware Requests resolved 2017-01-20 2016-09-14
141762 decommission snapshot1002, 1003, 1004 Hardware Requests resolved 2017-01-02 2016-10-13
142578 codfw/eqiad:(9+9) hardware access request for ORES Hardware Requests resolved 2017-01-02 2017-06-22
146455 Decommission labsdb1002 Hardware Requests resolved 2017-08-06 2017-03-03
147309 Decommission db1019 Hardware Requests resolved 2017-01-02 2017-04-12
148513 codfw/eqiad: 2x systems for prometheus Hardware Requests resolved 2017-08-02 2017-01-20
154706 Codfw: (1) hardware access request for labtestneutron refresh Hardware Requests resolved 2017-08-05 2017-06-14
158204 Eqiad: (2) hardware access request for labnet1003/1004 Hardware Requests resolved 2017-06-11 2017-05-19
161753 eqiad: (1) hardware access request for labnodepool1002 Hardware Requests resolved 2017-07-30 2017-06-27
164513 reclaim tempdb2001(WMF6407) to spares Hardware Requests resolved 2017-05-05 2017-05-11
164959 Decomission mw2098 Hardware Requests resolved 2017-07-31 2017-06-26
76306 Set warning thresholds for average cluster utilization Other Operations open 2017-04-21 2016-12-16
86081 Complete the use of HHVM over Zend PHP on the Wikimedia cluster Other Operations open 2017-07-21 2017-08-01
95054 Move ircecho config file to be YAML Other Operations open 2017-07-10 2016-07-15
97524 ocg alarm ocg_job_status_queue 'flapping' Other Operations open 2017-04-28 2016-10-03
97909 Upgrade jobrunners to redis 2.8 Other Operations open 2017-01-03 2016-02-13
101141 udp rcvbuferrors and inerrors on graphite1001 Other Operations open 2017-01-02 2017-07-10
104671 Rename 'restricted' group? Other Operations open 2017-07-12 2017-07-11
106937 Monitor [[Special:ListFiles]] for non 200 HTTP statuses in thumbnails Other Operations open 2017-01-02 2017-08-03
113104 Set up a service IP for logstash Other Operations open 2017-06-24 2017-06-23
115757 document debian packaging guidelines Other Operations open 2017-07-02 2016-05-23
116580 monitor postgresql replication status Other Operations open 2017-05-31 2015-10-26
116627 Include 5xx numbers in fluorine fatalmonitor Other Operations open 2017-08-08 2015-10-28
119401 Untangle labs/production roles from labs/instance roles Other Operations open 2017-07-03 2015-11-23
119846 Redirect revisions from svn.wikimedia.org to https://phabricator.wikimedia.org/rSVN Other Operations open 2017-01-02 2015-11-30
120165 Implement role based hiera lookups for labs Other Operations open 2017-02-27 2017-03-29
120943 Wikimania 2017 site does not automatically redirect to mobile site, when opening from a mobile device Other Operations open 2017-07-26 2017-07-25
121610 system users with UIDs > 500 Other Operations open 2017-07-09 2015-12-16
122210 Security audit for tftp on Carbon Other Operations open 2017-01-02 2016-03-29
122825 Service Ownership and Maintenance Other Operations open 2017-01-02 2017-08-08
124413 confctl should provide tags information after writing data Other Operations open 2017-02-11 2016-01-22
125442 es2009 degraded RAID Other Operations open 2017-07-22 2016-04-22
126158 [RFC] Alert about *when* partitions will run out of space, not a percentage/absolute number Other Operations open 2017-07-12 2016-05-02
129224 on labcontrol1001, /var/cache/salt has too many files! Other Operations open 2017-01-02 2016-03-08
129841 Many minions fail to connect to salt master since 10:39 Other Operations open 2017-07-17 2016-03-14
130617 Collect metrics on pool counter usage Other Operations open 2017-01-02 2016-11-03
133476 Proposal: Centralize OTRS login methodology Other Operations open 2017-07-20 2016-05-21
133656 Have a paging check for Nova API accessible Other Operations open 2017-05-22 2016-12-21
133913 Completely port l10nupdate to scap Other Operations open 2017-01-17 2017-03-06
134551 Create functional cluster checks for all services (and have them page!) Other Operations open 2017-01-02 2017-07-12
135124 Deploy etcddump (or another etcd dump & load tool) to production Other Operations open 2017-05-28 2016-05-12
135318 Document how to handle 'inconsistent state within the internal storage backends' issues Other Operations open 2017-01-02 2016-07-31
135338 On Trusty and Jessie PHP yields: PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/20-xhprof.ini on line 2 Other Operations open 2017-06-06 2017-07-01
136311 Monitor the BMC's event log for hardware errors Other Operations open 2017-01-02 2016-05-26
140442 reinstall rdb100[56] with RAID Other Operations open 2017-06-27 2017-04-18
140813 Protect sensitive user-related information with a UserData / auth / session service Other Operations open 2017-07-06 2017-06-08
141038 implement icinga paging for non-ops teams Other Operations open 2017-04-01 2016-07-21
141756 audit / test / upgrade hp smartarray P840 firmware Other Operations open 2017-04-22 2017-06-06
141897 Review new service 'pre-deployment to production' checklist Other Operations open 2017-01-02 2017-07-12
144539 Remove /srv/deployment/wdqs/wdqs/rules.log symlink Other Operations open 2017-01-02 2016-10-11
146664 Limit resources used by ORES Other Operations open 2017-02-15 2017-05-01
146914 grain-ensure erroneous mismatch with (bool)True vs (str)true Other Operations open 2017-01-02 2017-04-21
147366 Setup automated topk wide row reporting Other Operations open 2017-07-19 2017-07-18
147872 Rename rhodium to puppetmaster1003 Other Operations open 2017-07-12 2016-10-11
149617 Integrating MediaWiki (and other services) with dynamic configuration Other Operations open 2017-05-01 2017-08-08
150460 Configure maps cluster to send statsd metrics to the statsd endpoint in the same datacenter Other Operations open 2017-03-16 2016-11-10
150672 Provide a /parsoid directory on releases.wikimedia.org Other Operations open 2017-01-02 2017-05-20
150771 Secondary production Jenkins for CI Other Operations open 2017-01-02 2017-05-16
151045 Extending Yubico 2FA for production use (meta bug) Other Operations open 2017-01-02 2016-11-18
151049 Run systematic availability tests Other Operations open 2017-04-12 2016-11-18
151050 Proper documentation for Yubico 2FA for production use Other Operations open 2017-05-20 2016-11-18
152439 cronspam from labtestservices2001 /etc/dns-floating-ip-updater.py > /dev/null Other Operations open 2017-01-02 2016-12-05
152562 Port fundraising stats off Ganglia Other Operations open 2017-06-19 2017-06-27
153416 docker-engine pulled into our repositories only keeps the latest version Other Operations open 2017-06-06 2017-01-04
154915 Get rid of "import realm.pp" in manifests/site.pp Other Operations open 2017-01-10 2017-07-06
156136 Increase swift replication factor for accounts Other Operations open 2017-06-10 2017-01-24
158429 Switch to predictable network interface names? Other Operations open 2017-07-18 2017-06-07
159242 Segmentation fault creating thumbnail Other Operations open 2017-03-01 2017-02-28
159536 Puppet constantly trying to stop the already stopped puppetmaster process on Trusty Other Operations open 2017-08-10 2017-03-03
160101 Upgrade php5-json .deb to at least 1.3.8 Other Operations open 2017-06-20 2017-03-09
160158 Make disabled accounts visible in the corp mirror LDAP replica Other Operations open 2017-03-11 2017-03-28
160941 Improve SSH access information in onboarding documentation Other Operations open 2017-06-30 2017-03-30
161528 incident 20170323-wikibase did not trigger Icinga paging Other Operations open 2017-06-14 2017-03-27
161918 videoscalers (mw1168, mw1169) - high load / overheating Other Operations open 2017-04-01 2017-04-01
161920 logrotate for ruthenium Other Operations open 2017-04-01 2017-05-10
162029 Migrate all jessie hosts to Linux 4.9 Other Operations open 2017-04-04 2017-04-13
162123 Running swiftrepl is not puppetized Other Operations open 2017-08-04 2017-07-06
163288 Decide on /var/lib vs /home as locations of homedir for l10nupdate Other Operations open 2017-04-20 2017-04-19
164290 Set up external DNS record for wikitech-static Other Operations open 2017-06-07 2017-06-06
165173 rack/setup/install dumpsdata100[12] Other Operations open 2017-05-13 2017-07-06
165348 Check long-running screen/tmux sessions Other Operations open 2017-06-11 2017-07-10
165520 rack and setup wtp1025-1048 Other Operations open 2017-05-17 2017-07-12
165618 Audit / document reasons for not enabling HT? Other Operations open 2017-07-24 2017-05-17
165781 rack/setup/install labcontrol100[34] Other Operations open 2017-05-20 2017-07-12
165784 rack/setup/install labmon1002 Other Operations open 2017-05-20 2017-08-03
166038 Sync internal nutcracker package with Debian package Other Operations open 2017-05-23 2017-05-29
166322 spam from phabricator in labs Other Operations open 2017-05-26 2017-05-26
167412 host-vmem.erb is doing operations that make no sense Other Operations open 2017-07-04 2017-06-17
167820 rack/setup/install labweb100[12].wikimedia.org Other Operations open 2017-06-14 2017-07-14
167992 rack/setup/install new kafka nodes kafka-jumbo100[1-6] Other Operations open 2017-06-16 2017-08-09
168619 Degraded RAID on lvs3001 Other Operations open 2017-06-23 2017-07-19
169035 bast3002 sdb broken Other Operations open 2017-06-29 2017-07-03
169658 Improve database backups' coverage, monitoring and data recovery time (part 1) (tracking) Other Operations open 2017-07-05 2017-07-04
170120 Standardize on the "default" pod setup Other Operations open 2017-07-11 2017-07-10
170740 PuppetDB misbehaving on 2017-07-15 Other Operations open 2017-07-16 2017-07-19
171188 Move the main WMCS puppetmaster into the Labs realm Other Operations open 2017-08-03 2017-07-24
171623 Split up labstore external shelf storage available in codfw between labstore2001 and 2 Other Operations open 2017-07-26 2017-07-25
47877 During deployment old servers may populate new cache URIs Other Operations resolved 2017-06-15 2017-04-28
54556 lists.wikimedia.org has no SPF record Other Operations resolved 2017-01-02 2017-03-11
63442 Use gzip transfer encoding for SVG files from MediaWiki static assets Other Operations resolved 2017-06-20 2016-04-18
71604 acct (process and login accounting) fill up instances /var/ partition Other Operations resolved 2017-06-08 2017-06-07
76208 IRCEcho package should put files in /usr/bin, not /usr/ircecho/bin Other Operations resolved 2017-01-02 2016-07-15
78799 HHVM Jenkins job throw: Unable to set CoreFileSize to 8589934592: Operation not permitted (1) Other Operations resolved 2017-02-04 2016-01-13
81030 gdash reports for php/apache errors Other Operations resolved 2017-06-17 2016-01-11
84044 clean up non-working otrs email addresses Other Operations resolved 2017-04-22 2017-01-29
86633 Unify ::production / ::beta roles for *oid Other Operations resolved 2017-06-08 2017-06-07
86668 Make all ldap users have a sane shell (/bin/bash) Other Operations resolved 2017-05-16 2017-07-07
86887 Setup redis clusters in codfw Other Operations resolved 2017-01-21 2016-01-20
86890 Setup imagescalers cluster in codfw Other Operations resolved 2017-07-22 2016-01-20
87197 Error: Sysctl::Parameters[wikimedia base]: Could not evaluate: can't dup Symbol on deployment-pdf01 Other Operations resolved 2017-01-02 2017-06-07
88870 Prototype Monolog and rsyslog configuration to ship log events from MediaWiki to Logstash Other Operations resolved 2017-01-02 2016-09-01
89912 HHVM lock-ups Other Operations resolved 2017-05-20 2017-04-07
90883 mc2004 console is unreadable remotely Other Operations resolved 2017-05-03 2016-01-20
92011 setup & deploy rdb2001-2004 Other Operations resolved 2017-03-02 2016-01-20
92239 cron's puppet-run fails silently if apt-get update fails Other Operations resolved 2017-06-08 2017-06-07
92306 cp3011 hardware fault Other Operations resolved 2017-06-21 2016-03-24
92331 Delete stat1002:/a/squid/archive/blog Other Operations resolved 2017-01-02 2016-02-12
94611 Automate compiling service dependencies using production Jessie libraries Other Operations resolved 2017-01-02 2016-10-05
94822 Pybal RunCommand monitor doesn't work correctly on ubuntu trusty Other Operations resolved 2017-07-01 2016-01-20
97294 Turn off webrequest udp2log instances. Other Operations resolved 2017-03-01 2015-12-15
97758 SVG rendering with marker-element is different between librsvg and Inkscape Other Operations resolved 2017-07-03 2016-07-04
98183 labstore1002 issues while trying to reboot Other Operations resolved 2017-05-23 2017-02-15
99105 Kafka Broker disk usage is imbalanced Other Operations resolved 2017-04-02 2016-02-04
101377 LDAP TLS failing on some instances due to inconsistent state Other Operations resolved 2017-01-02 2017-06-07
114443 EventBus MVP Other Operations resolved 2017-01-02 2016-05-22
115505 Can't see any page, special:RandomPage gives database error Other Operations resolved 2017-01-02 2016-01-11
115965 [Regression] 404 Not Found: https://en.wikipedia.org/apple-touch-icon.png Other Operations resolved 2017-04-16 2016-01-14
116019 Rack/Setpup labvirt1010 and 1011 Other Operations resolved 2017-08-11 2015-10-27
116184 mw1083's sda disk is dying Other Operations resolved 2017-05-17 2015-11-11
116248 Off Boarding: Remove user pbeaudette from aliases Other Operations resolved 2017-02-02 2015-11-24
116487 Requesting access to (wmf-)deployment for jzerebecki Other Operations resolved 2017-03-15 2015-11-02
116673 puppetize wdqs service startup on boot Other Operations resolved 2017-05-12 2015-10-29
116963 upgrade radium to jessie Other Operations resolved 2017-07-08 2015-10-30
117394 Job queue broken for labswiki (jobs for wikitech.wikimedia.org are not running) Other Operations resolved 2017-03-26 2015-12-11
117477 reclaim lawrencium to spares Other Operations resolved 2017-08-08 2015-11-10
117994 Add monitoring for analytics-statsv service Other Operations resolved 2017-01-02 2016-11-09
118148 deployment broken on wdqs1001 Other Operations resolved 2017-01-02 2015-11-10
118495 uwsgi takes a long time to restart (Debian Jessie in labs) Other Operations resolved 2017-01-02 2016-08-12
118817 Wikimedia pageview API intermittently throwing HTTP 503s Other Operations resolved 2017-01-02 2015-11-17
118857 Investigate why mw2208 is powered off Other Operations resolved 2017-01-02 2015-12-01
118976 establish new thresholds for cassandra alarms after switching restbase to dtcs Other Operations resolved 2017-05-19 2016-12-19
119558 sort of SLA clarification grafana.wikimedia.org Other Operations resolved 2017-01-02 2015-12-01
119896 check for spare disk bays in restbase1007-1009 Other Operations resolved 2017-03-25 2015-12-01
120033 this is a test subtask Other Operations resolved 2017-01-02 2015-12-01
120523 Create a new mailing list: elf@lists.wikimedia.org Other Operations resolved 2017-01-02 2015-12-16
120672 New Onboard's (Marc Brent) Email not working Other Operations resolved 2017-02-02 2015-12-07
120731 eventlog1001 access for user Madhuvishy Other Operations resolved 2017-08-08 2015-12-14
120870 Clean up some accidental restbase metrics Other Operations resolved 2017-08-10 2016-02-18
121231 restbase should send metrics in batches Other Operations resolved 2017-06-01 2016-02-05
121492 fuse-dfs problems on stat1002 Other Operations resolved 2017-01-25 2016-08-08
121582 setup/deploy einsteinium as monitoring host Other Operations resolved 2017-03-27 2016-10-31
121594 Stale copy of Wikipedia:Featured picture candidates/Peacock butterfly and other mobile pages being served to users Other Operations resolved 2017-02-15 2016-01-20
122005 mw1228 reporting readonly file system Other Operations resolved 2017-06-24 2016-02-05
122062 Create the wikimania2017 wiki Other Operations resolved 2017-06-26 2015-12-28
122475 the centralauth databases is accessible form the mysql shell on terbium only in some cases Other Operations resolved 2017-01-02 2016-04-27
122909 puppet compiler runs fail when backup::host is included on host Other Operations resolved 2017-02-19 2016-02-24
123252 root shell for ema Other Operations resolved 2017-01-02 2016-01-12
123285 remove gbyrd from exim alias file Other Operations resolved 2017-07-12 2016-01-14
123315 URGENT - remove jimmy@ alias from exim mail aliases Other Operations resolved 2017-06-01 2016-01-11
123406 exim alias remove - feed-sales Other Operations resolved 2017-06-27 2016-01-12
123438 uwsgi puppet module does not seem to trigger restart when config is updated Other Operations resolved 2017-07-31 2016-01-16
123472 Site: (1) VM request for url_downloader Other Operations resolved 2017-07-23 2016-01-13
123521 [Bug] ValueView GitHub mirror not updated any more Other Operations resolved 2017-01-02 2016-02-22
123525 reduce amount of remaining Ubuntu 12.04 (precise) systems in production Other Operations resolved 2017-06-10 2017-03-29
123546 Possible bad mem chip or slot on dbproxy1004 Other Operations resolved 2017-01-02 2016-01-21
123562 remove exim alias - mkahn Other Operations resolved 2017-01-04 2016-01-14
123578 remove exim alias - corissa Other Operations resolved 2017-01-02 2016-01-14
123646 remove exim alias keynote@ Other Operations resolved 2017-07-08 2016-01-14
123681 add datacenter-ops to dhcp /install-server and allow to run puppet commands Other Operations resolved 2017-05-08 2016-01-25
123728 replace fluorine with mwlog servers (was: Upgrade fluorine to trusty/jessie) Other Operations resolved 2017-07-28 2017-07-25
123785 Requesting access to analytics-privatedata-users for jdcc-berkman Other Operations resolved 2017-01-27 2016-01-19
123796 Wes Moran not able to log into Graphite Other Operations resolved 2017-07-03 2016-01-20
124026 asw-c1-eqiad uplinks are down Other Operations resolved 2017-03-31 2016-01-19
124639 Metrics not reaching Graphite Other Operations resolved 2017-07-10 2016-07-21
125056 rhodium.eqiad.wmnet status? Other Operations resolved 2017-01-02 2016-01-30
125058 Prepare mathoid for the codfw switchover Other Operations resolved 2017-01-02 2016-03-16
125061 Prepare mobileapps for the codfw switchover Other Operations resolved 2017-01-02 2016-03-16
125069 Create a service location / discovery system for locating local/master resources easily across all WMF applications Other Operations resolved 2017-08-04 2017-02-01
125267 ~3000% increase in session redis memory usage, causing evictions and session loss Other Operations resolved 2017-03-23 2016-01-30
125565 Update Label for oresrdb1001 (WMF4577) & relocate and update label for oresrdb1002 (WMF4578) Other Operations resolved 2017-07-16 2016-02-12
126221 Evaluate efficacy of DateTieredCompactionStrategy Other Operations resolved 2017-01-02 2017-07-11
126242 Reduce the number of appservers we're using in eqiad Other Operations resolved 2017-08-01 2016-04-27
126262 Install php5-readline on trusty and jessie hosts so eval.php, sql.php, and so on are more useful Other Operations resolved 2017-01-28 2016-04-14
126283 Requesting restbase-roots access to RESTBase cluster for Petr Pchelko Other Operations resolved 2017-01-02 2016-03-01
126447 add more statsdlb instances for more throughput Other Operations resolved 2017-01-02 2016-07-28
126586 elastic1023 doesn't come back up after reboot Other Operations resolved 2017-03-26 2016-02-11
127053 iridium / phabricator not accepting email via smtp Other Operations resolved 2017-05-15 2016-02-18
127189 Andrew gets pages many hours too late Other Operations resolved 2017-01-02 2016-02-17
127488 (re)move problemsdonating aliases Other Operations resolved 2017-01-02 2017-07-07
127489 donation aliases for moneybookers? Other Operations resolved 2017-07-16 2016-03-04
127493 remove / change blog@wm mail alias? Other Operations resolved 2017-07-19 2016-02-19
127720 Allow aqs-admins to deploy via scap using deploy-service ssh key Other Operations resolved 2017-04-02 2016-02-22
127808 access for nikerabbit to researchers Other Operations resolved 2017-04-02 2016-02-26
128057 db2018 failed disk (degraded RAID) Other Operations resolved 2017-04-04 2016-09-22
128369 ganglia cluster name validation in puppet Other Operations resolved 2017-05-15 2016-03-01
128463 New Service Request - Change Propagation Other Operations resolved 2017-01-02 2016-04-01
128512 Reset mailing list admin password for wikimedia-dz Other Operations resolved 2017-03-13 2016-03-03
128639 Remove rbraceysherman@ from fr-all list Other Operations resolved 2017-07-17 2016-03-02
128845 No serial console on iron's mgmt interface Other Operations resolved 2017-03-01 2016-03-04
129062 Reorganize mw2* servers across clusters Other Operations resolved 2017-01-02 2016-03-07
129152 Deploy servermon with scap3 Other Operations resolved 2017-01-12 2017-06-06
129168 fr-all fails with error 451 Other Operations resolved 2017-04-07 2016-03-08
129222 Icinga disk space check should also check inode usage Other Operations resolved 2017-06-18 2017-07-27
129260 Requesting access to to analytics-search-user for Mikhail Popov and Oliver Keyes Other Operations resolved 2017-04-19 2016-03-16
129517 The refreshLinks jobs enqueue rate is 10 times the normal rate Other Operations resolved 2017-01-22 2017-05-05
129586 /mnt/upload7 does not exist anywhere, yet it is referenced in multiple places in wmf-config Other Operations resolved 2017-01-02 2016-09-12
129623 labnet1002 can't talk to webproxy.eqiad.wmnet:8080, puppet fails to install designateclient Other Operations resolved 2017-01-02 2016-04-11
129973 Dig through logs from 15 Mar 2016 read-only test and file bugs Other Operations resolved 2017-01-02 2016-05-10
130174 elastic1005.eqiad.wmnet is non-responsive Other Operations resolved 2017-06-24 2016-03-17
130226 stat1001 access + sudo rights for nuria and mforns Other Operations resolved 2017-03-06 2016-03-22
130575 Multiple users reporting content pages displaying "NULL" compared to desired content Other Operations resolved 2017-03-22 2016-03-22
130990 restbase2004.codfw.wmnet: Failed disk/RAID Other Operations resolved 2017-01-02 2016-04-08
131147 ms-be2008.codfw.wmnet: slot=1 dev=sdl failed Other Operations resolved 2017-05-08 2016-06-15
131150 Add monitoring metric for connection tracking table usage Other Operations resolved 2017-05-06 2016-04-20
131195 Bacula recovery of sql files from silver/wikitech fails Other Operations resolved 2017-08-07 2016-04-20
131746 Integrate jessie 8.4 point release Other Operations resolved 2017-01-02 2017-02-07
131877 setup stat1004/WMF4721 for hadoop client usage Other Operations resolved 2017-04-02 2016-04-19
132033 Backport libvpx2 to jessie Other Operations resolved 2017-06-24 2016-04-20
132422 cronspam from labscontrol1001, labstore1001, labnet1002.eqiad.wmnet, labsdb1003.eqiad.wmnet Other Operations resolved 2017-05-18 2016-10-05
132751 "Elastica: missing curl_init_pooled method" due to mwscript job running with PHP 5 on terbium Other Operations resolved 2017-04-27 2017-03-17
132757 Split carbon's install/mirror roles, provision install1001 Other Operations resolved 2017-01-30 2017-02-14
132924 Remove cache files from Planet to regenerate fr.planet.wikimedia.org page Other Operations resolved 2017-01-02 2016-04-25
132952 Failed to deploy WDQS Other Operations resolved 2017-04-02 2017-07-14
132976 rack/setup/deploy restbase200[7-9] Other Operations resolved 2017-06-12 2016-06-17
133047 Write a test to check for clearly bogus hostnames Other Operations resolved 2017-01-02 2016-06-17
133048 BadMethodCallException when viewing Items created around the time of the eqiad -> codfw switch Other Operations resolved 2017-01-02 2016-04-19
133785 rack/setup/deploy aqs100[456] Other Operations resolved 2017-01-02 2016-06-17
133966 missing /etc/ssl/dhparam.pem on jessie Apache using ssl_ciphersuite Other Operations resolved 2017-07-13 2016-04-29
133992 Allow RelEng access to labnet servers (was: Allow RelEng nova log access) Other Operations resolved 2017-01-02 2017-06-27
134197 plug in restbase2004 network cable Other Operations resolved 2017-01-02 2016-05-25
134241 graphoid should not use the http proxy to connect to the mediawiki api and other internal services Other Operations resolved 2017-01-02 2016-06-17
134419 repair/replace pem1 in cr1-ulsfo Other Operations resolved 2017-08-10 2016-05-05
134485 Package Thumbor for Debian Other Operations resolved 2017-01-02 2016-07-25
134866 Automatic monitoring checks for the MCS failing in production Other Operations resolved 2017-01-02 2016-09-23
135178 investigate RAID failure on beryllium.frack.eqiad.wmnet Other Operations resolved 2017-01-02 2016-06-01
135386 Cleanup puppet from unneeded and empty single-service "roots" Other Operations resolved 2017-03-09 2016-06-17
135557 kafka1013 hardware crash Other Operations resolved 2017-06-17 2016-07-27
135776 Reset Mailman List Creator password Other Operations resolved 2017-01-02 2016-05-19
136405 Jmxtrans failures on Kafka hosts caused metric holes in grafana Other Operations resolved 2017-04-02 2016-08-03
136708 rack/upgrade/setup/install/deploy relforge100[12].eqiad.wmnet Other Operations resolved 2017-01-02 2016-06-07
137851 "Is maps service alive?" check Other Operations resolved 2017-06-11 2016-06-17
137878 Allow *-admin groups to see systemd logs for their units Other Operations resolved 2017-01-02 2016-08-10
138415 eqiad: Install ssds to labmon1001 Other Operations resolved 2017-04-08 2016-06-27
138627 Enable WDQS admins to enable/disable mask/unmask updater service Other Operations resolved 2017-05-29 2016-07-06
138882 Monitor production worker nodes in icinga Other Operations resolved 2017-01-02 2016-06-29
139202 access: eventlogging-admins -> hafnium Other Operations resolved 2017-01-02 2016-07-11
139234 Phabricator XSS vulnerability Other Operations resolved 2017-06-09 2016-07-13
139362 High storage utilization on restbase1014.eqiad.wmnet Other Operations resolved 2017-01-02 2016-07-09
139728 Create a Goal project for technical operations Q1 goal "Modernize Puppet configuration management infrastructure" Other Operations resolved 2017-05-25 2016-08-08
139946 Enable Interactive Maps (Kartographer) on Macedonian Wikipedia Other Operations resolved 2017-01-02 2016-09-04
140264 MediaWiki deployment shell access request for zfilipin Other Operations resolved 2017-01-02 2016-07-18
140598 db2056 RAID controller (temporary) failure Other Operations resolved 2017-05-21 2016-07-18
140646 deploy prometheus node_exporter for host monitoring Other Operations resolved 2017-07-11 2016-11-03
140886 Unable to access Etherpad - https://etherpad.wikimedia.org/p/Fundraising_Staff_Feedback Other Operations resolved 2017-05-26 2016-08-10
140899 Move labs graphite to graphite-labs.wikimedia.org Other Operations resolved 2017-06-27 2016-07-21
141242 Make the WMF puppet tree compile equally under puppet 3.4 and 3.8 Other Operations resolved 2017-05-23 2016-07-26
141360 stat1004 doesn't show up in ganglia Other Operations resolved 2017-04-02 2016-08-01
141522 Physically decommission mw1001-mw1148 (except mw1017 and mw1099) Other Operations resolved 2017-05-30 2017-04-27
141634 Analytics cluster access request for ISI Foundation team Other Operations resolved 2017-01-02 2016-08-18
141957 ferm rules on icinga are broken, Other Operations resolved 2017-08-10 2016-08-03
142216 Enable root passwords on Labs VMs Other Operations resolved 2017-01-02 2016-09-01
142309 Some users can't login or edit without proxy in Iran Other Operations resolved 2017-01-02 2016-11-16
143465 Access to people.wikimedia.org for Volker_E Other Operations resolved 2017-07-18 2017-02-08
143718 Request access to data for reader research Other Operations resolved 2017-01-02 2016-08-29
144380 Install and configure new WDQS nodes on codfw Other Operations resolved 2017-04-02 2016-12-22
144389 /a/mw-log/hhvm.log: file not found on Fluorine Other Operations resolved 2017-05-16 2016-08-31
144396 Disable cron job to clear elasticsearch caches and validate that it does not have significant impact on GC Other Operations resolved 2017-05-29 2016-09-05
144416 reset password of winedale-l Other Operations resolved 2017-06-08 2016-09-01
144469 onboarding Manuel Arostegui in ops Other Operations resolved 2017-02-16 2016-09-01
144502 deploy prometheus node_exporter and server to deployment-prep Other Operations resolved 2017-01-02 2016-10-07
144537 Make puppet generate path config for WDQS nodes Other Operations resolved 2017-01-02 2016-09-12
144624 Requesting access to stat1002 for ZZhou (WMF) Other Operations resolved 2017-01-02 2016-09-08
144793 db1020 degraded array Other Operations resolved 2017-08-10 2016-09-07
144801 Icinga check for unaccepted Salt keys Other Operations resolved 2017-05-18 2016-09-17
144928 thumbor/exiftool deadlock, likely full pipe Other Operations resolved 2017-01-02 2016-09-13
145618 Thumbor SVG regexp insufficient Other Operations resolved 2017-01-02 2016-10-05
145626 thumbor ffmpeg pipe deadlock Other Operations resolved 2017-01-02 2016-10-06
145632 Separate 404s into their own log Other Operations resolved 2017-07-26 2016-10-18
145914 access request for 'researchers' and 'analytics-users' for Deborah Tankersley Other Operations resolved 2017-03-28 2016-09-21
146158 Configure varnish to include wdqs nodes in codfw Other Operations resolved 2017-05-03 2016-10-25
146464 hhvm root:adm owned log files cause failures for logrotate Other Operations resolved 2017-05-06 2017-06-12
147181 Docker installation for production kubernetes Other Operations resolved 2017-04-14 2016-10-24
147397 Upgrade OTRS to 5.0.13 Other Operations resolved 2017-05-15 2016-10-11
147918 Log thumbnail requests that fail on Thumbor and not on Mediawiki and vice versa Other Operations resolved 2017-01-31 2016-11-14
148361 SVG and VIPS engines expect temp folder to stay a while when it doesn't Other Operations resolved 2017-01-02 2016-10-17
148966 Puppet implementation of the production docker-registry installation Other Operations resolved 2017-01-02 2016-11-02
149078 Upgrade firejail to 0.44 Other Operations resolved 2017-07-13 2017-02-14
149228 Rename stat100x machines to have misc element names Other Operations resolved 2017-01-02 2017-06-05
149234 RAID degraded on ms-be2011 Other Operations resolved 2017-02-13 2016-11-23
149432 puppet compiler claims "no change" when catalogs are actually different Other Operations resolved 2017-01-02 2017-08-04
149981 Ask firejail upstream about ability to turn off pid namespacing Other Operations resolved 2017-07-30 2016-11-07
150043 New Service Request for Trending Edits Service Other Operations resolved 2017-01-02 2017-01-27
150316 decom gallium (data center) Other Operations resolved 2017-01-02 2017-05-03
150490 decom neon (data center) Other Operations resolved 2017-01-02 2017-03-03
150649 Thumbor keeps losing Swift auth on beta Other Operations resolved 2017-03-16 2016-11-17
150754 Thumbor can't render a few SVGs that Mediawiki can Other Operations resolved 2017-01-02 2017-01-20
150760 Thumbor 404s when the original has a ? in its filename Other Operations resolved 2017-01-02 2016-12-02
151338 rack/setup prometheus200[3-4] Other Operations resolved 2017-06-14 2016-12-27
151427 mw2092 - disk issue Other Operations resolved 2017-01-02 2016-12-01
151441 Thumbor should handle "temp" thumbnail requests Other Operations resolved 2017-01-02 2017-02-01
151456 Thumbor original file download limit should be 4GB Other Operations resolved 2017-05-04 2017-03-28
152023 adrian bielefeldt & julius gonsior shell request + analytics-privatedata-users Other Operations resolved 2017-01-02 2016-12-09
152473 Upgrade Grafana to 4.0.2 Other Operations resolved 2017-06-27 2017-03-02
152643 rack/setup/install wdqs1003 Other Operations resolved 2017-06-10 2017-02-06
153278 Trying to scap while l10nupdate is syncing shows unhelpful error Other Operations resolved 2017-07-20 2017-02-10
154927 Deploy TwoColConflict extension to beta Other Operations resolved 2017-08-07 2017-06-07
155181 codfw:mw2251-mw2260 switch port configuration Other Operations resolved 2017-06-29 2017-01-18
155443 Update ci to nodejs 6 Other Operations resolved 2017-01-17 2017-03-30
156151 Cronspam from mwlog* Other Operations resolved 2017-02-26 2017-03-15
156154 ship R620 power supplies to ulsfo Other Operations resolved 2017-06-18 2017-01-24
156529 fix log reading permissions for dc-ops admin group Other Operations resolved 2017-05-28 2017-02-07
157483 Request for access to stat1003 for Sam Tarling Other Operations resolved 2017-04-25 2017-02-15
157797 Run Icinga check for systemd-timedated less often Other Operations resolved 2017-02-11 2017-02-14
158131 hard-reset DRAC gadolinium.mgmt.eqiad.wmnet Other Operations resolved 2017-03-21 2017-04-27
158176 Build / migrate to HHVM 3.18 Other Operations resolved 2017-04-16 2017-06-15
158337 codfw: ms-be2028-ms-be2039 rack/setup Other Operations resolved 2017-03-03 2017-04-18
158664 elasticsearch logs are duplicated in journald Other Operations resolved 2017-02-22 2017-05-30
158789 labmon1001: Graphite + Mirantis' Django don't play well together Other Operations resolved 2017-02-23 2017-02-22
159168 Add elasticsearch 5 .deb to reprepro experimental repository Other Operations resolved 2017-04-24 2017-03-31
159486 setup/deploy scb2005 & scb2006 Other Operations resolved 2017-05-11 2017-04-28
159887 rack and cable frdev1001 Other Operations resolved 2017-03-08 2017-04-04
160082 codfw: oresrdb2002 rack/setup Other Operations resolved 2017-03-10 2017-03-13
161158 Degraded RAID on ocg1001 Other Operations resolved 2017-03-23 2017-05-02
161193 Upgrade to Grafana 4.2.0 Other Operations resolved 2017-03-24 2017-03-27
161520 HHVM 3.18 crashes when Cirrus tries to fetch another wiki config via maint script Other Operations resolved 2017-08-09 2017-04-05
162216 rack and set up analyics1058-1069 Other Operations resolved 2017-04-08 2017-06-14
162770 SATA errors for stat1004 in the dmesg Other Operations resolved 2017-04-13 2017-07-27
162949 hosts with puppet compiler failures on every run Other Operations resolved 2017-04-14 2017-08-09
163196 Puppet facts around the primary network interface and IPv4/IPv6 address Other Operations resolved 2017-04-19 2017-05-16
163408 (icinga/grafana) webpagetest-alerts: slow page rendering for Internet Explorer Other Operations resolved 2017-05-20 2017-05-03
163721 Update wikitech-static and develop procedures to keep it maintained Other Operations resolved 2017-08-09 2017-05-11
163889 cp1066 troubleshoot IPMI issue Other Operations resolved 2017-04-27 2017-04-27
163942 Ops pwstore currently read-only Other Operations resolved 2017-04-27 2017-04-27
165051 HHVM 3.18 segfault on jobrunner / string handling Other Operations resolved 2017-07-06 2017-06-09
165220 Degraded RAID on labstore1003 Other Operations resolved 2017-05-14 2017-05-30
165467 Create an etcd cluster in codfw for kubernetes usage Other Operations resolved 2017-05-22 2017-05-17
165733 Supersede RT tickets references Other Operations resolved 2017-05-20 2017-06-16
165943 Access to search logs for Jan Dittrich Other Operations resolved 2017-08-05 2017-05-30
166345 wmf/1.30.0-wmf.2 performance issue for Wikipedias Other Operations resolved 2017-05-26 2017-06-07
166552 Switch etcd back to eqiad, document switchover procedure Other Operations resolved 2017-05-31 2017-05-31
166776 Geoshape and geoline subservices need monitoring Other Operations resolved 2017-06-02 2017-06-20
166806 Server side upload for Yann Other Operations resolved 2017-06-24 2017-06-06
167157 rack/setup/install labtestpuppetmaster2001 Other Operations resolved 2017-06-28 2017-08-07
167871 Refactor maps puppet code to the role / profile paradigm Other Operations resolved 2017-07-20 2017-06-22
167878 mwlog1001.eqiad.wmnet ‘/srv/mw-log/hhvm.log’ for reading: No such file or directory Other Operations resolved 2017-06-15 2017-06-22
168444 Logo for sr.wikiquote.org Other Operations resolved 2017-07-14 2017-07-14
168782 Create fishbowl wiki for Maithili Wikimedians User Group Other Operations resolved 2017-07-13 2017-07-18
168892 rack/setup/install labtestservices2002.wikimedia.org Other Operations resolved 2017-06-27 2017-07-24
169360 Unresponsive/misconfigured iDRACs over the host-BMC interface Other Operations resolved 2017-07-01 2017-07-12
169485 Add support for directory environments to our puppet classes, production puppetmaster Other Operations resolved 2017-08-10 2017-07-10
169871 mgmt inaccessible on restbase1018 Other Operations resolved 2017-07-23 2017-07-12
169959 bast3002 didn't come up after reboot Other Operations resolved 2017-07-09 2017-07-07
170653 create netmon1003, migrate servermon from netmon1001 to netmon1003 Other Operations resolved 2017-07-15 2017-07-15
171280 wikitech api list=novainstances not returning list of instances Other Operations resolved 2017-07-22 2017-07-21
172689 Install missing Spamassassin DKIM dependencies on lists and mx Other Operations resolved 2017-08-08 2017-08-07