Discussion:
buildbot.python.org down again?
Ned Deily
2014-07-06 23:54:50 UTC
Permalink
As of the moment, buildbot.python.org seems to be down again. Where is
the best place to report problems like this?
--
Ned Deily,
***@acm.org
Terry Reedy
2014-07-07 06:33:04 UTC
Permalink
Post by Ned Deily
As of the moment, buildbot.python.org seems to be down again.
Several hours later, back up.
Post by Ned Deily
Where is the best place to report problems like this?
We should have, if not already, an automatic system to detect down
servers and report (email) to appropriate persons.
--
Terry Jan Reedy
Guido van Rossum
2014-07-07 15:44:28 UTC
Permalink
It would still be nice to know who "the appropriate persons" are. Too much
of our infrastructure seems to be maintained by house elves or the ITA.
Post by Terry Reedy
Post by Ned Deily
As of the moment, buildbot.python.org seems to be down again.
Several hours later, back up.
Post by Ned Deily
Where is the best place to report problems like this?
We should have, if not already, an automatic system to detect down servers
and report (email) to appropriate persons.
--
Terry Jan Reedy
_______________________________________________
Python-Dev mailing list
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/
guido%40python.org
--
--Guido van Rossum (python.org/~guido)
Benjamin Peterson
2014-07-07 15:55:50 UTC
Permalink
Post by Guido van Rossum
It would still be nice to know who "the appropriate persons" are. Too much
of our infrastructure seems to be maintained by house elves or the ITA.
:) Is ITA "International Trombone Association"?
Guido van Rossum
2014-07-07 17:22:21 UTC
Permalink
It's a reference to Neil Stephenson's Anathem.
Post by Benjamin Peterson
Post by Guido van Rossum
It would still be nice to know who "the appropriate persons" are. Too much
of our infrastructure seems to be maintained by house elves or the ITA.
:) Is ITA "International Trombone Association"?
Antoine Pitrou
2014-07-07 17:47:31 UTC
Permalink
Post by Guido van Rossum
It's a reference to Neil Stephenson's Anathem.
According to Google, it doesn't look like he played the trombone, though.

Regards

Antoine.
Post by Guido van Rossum
Post by Guido van Rossum
It would still be nice to know who "the appropriate persons" are. Too much
of our infrastructure seems to be maintained by house elves or
the ITA.
:) Is ITA "International Trombone Association"?
Nick Coghlan
2014-07-08 04:58:50 UTC
Permalink
Post by Guido van Rossum
It would still be nice to know who "the appropriate persons" are. Too
much of our infrastructure seems to be maintained by house elves or the ITA.

I volunteered to be the board's liaison to the infrastructure team, and
getting more visibility around what the infrastructure *is* and how it's
monitored and supported is going to be part of that. That will serve a
couple of key purposes:

- making the points of escalation clearer if anything breaks or needs
improvement (although "***@python.org" is a good default choice)
- making the current "todo" list of the infrastructure team more visible
(both to calibrate resolution time expectations and to provide potential
contributors an idea of what's involved)

Noah has already set up http://status.python.org/ to track service status,
I can see about getting buildbot.python.org added to the list.

Cheers,
Nick.
Post by Guido van Rossum
Post by Terry Reedy
Post by Ned Deily
As of the moment, buildbot.python.org seems to be down again.
Several hours later, back up.
Post by Ned Deily
Where is the best place to report problems like this?
We should have, if not already, an automatic system to detect down
servers and report (email) to appropriate persons.
Post by Guido van Rossum
Post by Terry Reedy
--
Terry Jan Reedy
_______________________________________________
Python-Dev mailing list
https://mail.python.org/mailman/listinfo/python-dev
https://mail.python.org/mailman/options/python-dev/guido%40python.org
Post by Guido van Rossum
--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
https://mail.python.org/mailman/listinfo/python-dev
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
Donald Stufft
2014-07-08 07:33:32 UTC
Permalink
It would still be nice to know who "the appropriate persons" are. Too much of our infrastructure seems to be maintained by house elves or the ITA.
- making the current "todo" list of the infrastructure team more visible (both to calibrate resolution time expectations and to provide potential contributors an idea of what's involved)
Noah has already set up http://status.python.org/ to track service status, I can see about getting buildbot.python.org added to the list.
Cheers,
Nick.
We (the infrastructure team) were actually looking earlier about
buildbot.python.org and we’re not entirely sure who "owns" buildbot.python.org.
Unfortunately a lot of the *.python.org services are in a similar state where
there is no clear owner. Generally we've not wanted to just step in and take
over for fear of stepping on someones toes but it appears that perhaps
buildbot.p.o has no owner?

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Guido van Rossum
2014-07-08 14:48:39 UTC
Permalink
May the true owner of buildbot.python.org stand up!

(But I do think there may well not be anyone who feels they own it. And
that's a problem for its long term viability.)

Generally speaking, as an organization we should set up a process for
managing ownership of *all* infrastructure in a uniform way. I don't mean
to say that we need to manage all infrastructure uniformly, just that we
need to have a process for identifying and contacting the owner(s) for each
piece of infrastructure, as well as collecting other information that
people besides the owners might need to know. You can use a wiki page for
that list for all I care, but have a process for what belongs there,
how/when to update it, and even an owner for the wiki page! Stuff like this
shouldn't be just in a few people's heads (even if they are board members)
nor should it be in a file in a repo that nobody has ever heard of.
Post by Nick Coghlan
Post by Guido van Rossum
It would still be nice to know who "the appropriate persons" are. Too
much of our infrastructure seems to be maintained by house elves or the ITA.
I volunteered to be the board's liaison to the infrastructure team, and
getting more visibility around what the infrastructure *is* and how it's
monitored and supported is going to be part of that. That will serve a
- making the points of escalation clearer if anything breaks or needs
- making the current "todo" list of the infrastructure team more visible
(both to calibrate resolution time expectations and to provide potential
contributors an idea of what's involved)
Noah has already set up http://status.python.org/ to track service
status, I can see about getting buildbot.python.org added to the list.
Cheers,
Nick.
We (the infrastructure team) were actually looking earlier about
buildbot.python.org and we’re not entirely sure who "owns"
buildbot.python.org.
Unfortunately a lot of the *.python.org services are in a similar state where
there is no clear owner. Generally we've not wanted to just step in and take
over for fear of stepping on someones toes but it appears that perhaps
buildbot.p.o has no owner?
-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
--
--Guido van Rossum (python.org/~guido)
Martin v. Löwis
2014-07-10 07:41:10 UTC
Permalink
May the true owner of buildbot.python.org <http://buildbot.python.org>
stand up!
Well, I think that's me (atleast by my definition of "true owner").
I requested that the machine be set up, and I deployed the software
that is running the service (it was also me who originally introduced
buildbot to the Python project).

On the other hand, I'm not at all "in charge" of that infrastructure
piece. I haven't logged into the machine in many months, and it's
Antoine who currently maintains its configuration. So I don't want to
be pinged when the machine is down.
(But I do think there may well not be anyone who feels they own it. And
that's a problem for its long term viability.)
I don't think that's actually the case for "ownership". But then, I also
think that ownership is not a very important concept for pydotorg. Most
owners will likely agree that they lose their right to have a say in it
when they stop maintaining the piece that they own.
Generally speaking, as an organization we should set up a process for
managing ownership of *all* infrastructure in a uniform way. I don't
mean to say that we need to manage all infrastructure uniformly, just
that we need to have a process for identifying and contacting the
owner(s) for each piece of infrastructure, as well as collecting other
information that people besides the owners might need to know. You can
use a wiki page for that list for all I care, but have a process for
what belongs there, how/when to update it, and even an owner for the
wiki page!
Unfortunately, that plan keeps failing. Everybody agrees that such a
list would be useful, so everybody makes their own list. I was
maintaining such a list in the Python wiki for some time, until a
board member decided that a publically-visible inventory is not
appropriate, and it must be a password-protected wiki - where I now
keep forgetting where the wiki is, in the first place, let alone
remembering how to log in.

Regards,
Martin
Ned Deily
2014-07-12 01:04:14 UTC
Permalink
Post by Donald Stufft
Post by Nick Coghlan
Post by Guido van Rossum
It would still be nice to know who "the appropriate persons" are. Too
much of our infrastructure seems to be maintained by house elves or the ITA.
I volunteered to be the board's liaison to the infrastructure team, and
getting more visibility around what the infrastructure *is* and how it's
monitored and supported is going to be part of that. That will serve a
- making the points of escalation clearer if anything breaks or needs
- making the current "todo" list of the infrastructure team more visible
(both to calibrate resolution time expectations and to provide potential
contributors an idea of what's involved)
Noah has already set up http://status.python.org/ to track service status,
I can see about getting buildbot.python.org added to the list.
We (the infrastructure team) were actually looking earlier about
buildbot.python.org and we're not entirely sure who "owns"
buildbot.python.org.
Unfortunately a lot of the *.python.org services are in a similar state where
there is no clear owner. Generally we've not wanted to just step in and take
over for fear of stepping on someones toes but it appears that perhaps
buildbot.p.o has no owner?
In parallel to this discussion, I ran into Noah at a meeting the other
day and we talked a bit about buildbot.python.org. As Donald noted, it
sounds like he and the infrastructure team are willing to add it to the
list of machines they monitor and reboot, as long as they wouldn't be
expected to administer the buildbot master itself. I checked with
Antoine and Martin and they are agreeable with that. So I think there
is general agreement that the infrastructure team can take on uptime
monitoring and rebooting of buildbot.python.org and that Antoine/Martin
would be the primary/secondary contacts/owners for other administrative
issues. Martin would also be happy if the infrastructure team could
handle installing routine security fixes as well. I'll leave it to the
interested parties to discuss it further among themselves.
--
Ned Deily,
***@acm.org
Loading...