POSE/OpenFF - Meeting recording by Fireflies.ai

EEthan Holz

David, it was fascinating. I was looking at the pricing between AWS and Azure today, and their pricing for the same gpu's are exactly the same, but the cpu and ram specs are completely different.

DDavid Swenson

Interesting.

EEthan Holz

For example, if you get a v 100 on AWS, you'll get eight cores and I don't know, let's say this is not accurate, but 64 gigs. But if you did the same on Azure, you'd get six cores but 84 gigs. So they give you just way more memory on Azure, but not as much cpu compute, which I think is kind of interesting.

DDavid Swenson

This is one of the reasons I'm interested in actually benchmarking was. Hey Jeff, right there with you.

JJeffrey Wagner

Hey David, sorry. I tried to start the zoom call with my computer close and it got very sad.

DDavid Swenson

We're waiting on Matt, is that right?

JJeffrey Wagner

Correct.

JJeffrey Wagner

Open.

DDavid Swenson

Jeff, are you aware of anybody having done cloud focused benchmarks with open? Mm. Because as were just discussing, like you can have the same gpu but different system setups because different cloud providers kind of give you a different default. And we thought it might be a little bit interesting to see if that led to any variants and to really give a price to performance estimate for people.

JJeffrey Wagner

Yeah, I'm imagining, let's see. Hopefully open eye has done it, but wouldn't have published the results. I think the problem is people who would have done these studies would have been people who were financially incentivized to do the studies. And those are basically people who own clouds that they want you to run OpenMm on. And at that point those benchmarks are probably so gamed that they might not be useful.

DDavid Swenson

I mean, again, it's not. This is. I don't think this is an expensive thing to do. It's one of the reasons we're thinking about doing it, because you don't need to run, you know, days of compute to get an estimate of performance. I have to see what you know. I know that Peter Eastman has like a benchmark set. I'll have to see how long they run it for. But you know, you need to run it long enough that it's kind of settled in, but you don't really need to run it long enough to do something scientifically valuable in order to get the. So I'm thinking we can probably get it in a few hours per gpu that we're interested in, which then we're talking about a few bucks for most of them.

JJeffrey Wagner

Yeah, it'd be interesting to check out. Yeah, that'd be interesting.

JJeffrey Wagner

Cool.

JJeffrey Wagner

I can start some meeting notes if we want, or if you guys have a place planned, we can do that.

DDavid Swenson

There don't have a place planned. I brought in fireflies, although it took your name since it's your meeting, but that way I'll have those notes after the fact.

JJeffrey Wagner

Okay, well, let's do that. And I'll just record and then I won't worry about notes too much. Today. Do I want to start a notes document anyway? No, let's yellow this. This will be fun. Okay, so we. Yeah, we're interested to try out the GitHub actions runners. And the first place that we think it would make sense for us would be on open ff evaluator. We have some basic tests for evaluator, but we would love to do tests that run a short simulation, make sure that approximately the same numbers come out every time. That would have been impractical without a GPU. We want to get started. The place that we're starting from is that we have a root account, I believe, with payment details hooked up for Openforce field on AWS.

JJeffrey Wagner

But I'm not sure if we just want to have stuff right on that root account, if we should make personal accounts and hook up our CI to that, or if there's another kind of account it sounded like maybe we should be making. So my goal today would be tell us what to do for account setup, and then if we have extra time, I'd love to see more hands on in detail what to do.

DDavid Swenson

Yeah. So I'll. I'll go ahead and share my not yet logged in AWS thing here, just so you can sort of follow along as I'm doing this. So, no, I'm actually not going to use Desmond. So I'm going to log in as root, which is not what I would usually do, but I actually, it is what I do with this account frequently because it gives me my. The root account is the only one that can look at the billing information and I want to regularly check and make sure that we're not spending too much money. So, you know, and I've got my whole two factor going, so I have to pull out that thing. All right. That helps if I didn't click. No, it's just changed. 877674. Okay. So also from the root account, we can see everything.

DDavid Swenson

And yes, we spent a $3 so far this month. Not bad.

EEthan Holz

Almost bankrupt. I know. This month. There we go. I did my job correctly.

DDavid Swenson

So there are two. Well, three things you'll have to do. The first thing you're going to want to do as soon as you start doing this is go ahead and go to your service quotas, because you're not going to have EC two services. You're going to have EC two instances for what you want. So you go to the EC two type of quota and then you're going to want either. You're definitely going to want on demand to start with. And you're going to want either P instances or G instances. Okay, wait, we now have the p is a better gpu. G is g. What is G again, Ethan? It's so it's not performance.

EEthan Holz

Yeah, it's. They're more like, they have. They're more well rounded gpu's. So you're, your P series are going to think like your v. In this case, the K 80s are the other ones. G instances have more of like a well rounded feature set. So think T four s l. Like smaller, more efficient gpu's that maybe have a cut down feature set. But like, most like, it has most of your compute capability that you need. So they're. They're accelerated computing, but they're not like HPC optimized or they're not as like built for large vRam instances. So if you need like a decent amount of compute, generally your best price is the G instances.

EEthan Holz

So, like, you can get some like, T four gpu's at like $0.50 an hour, whereas, like, the cheapest on a P instance is going to be like closer to a dollar an hour. It's like $0.90.

JJeffrey Wagner

Yeah.

EEthan Holz

So that's kind of the difference. Difference there.

DDavid Swenson

But in any case, you will want to go through the process of going.

JJeffrey Wagner

On, how do I get to this page? Or let's establish this. I would love to follow along on my own computer. I'm on two pages ago, but then I lost track of you.

DDavid Swenson

Okay, so if we go back here ish. So we've got console. The first thing you do is you just. Yep, I can do that. Let me. So service quotas on AWS services. Type ec two. Just type. Yeah. And that's the one you want. Amazon Ec two, elastic cloud computer quotas. And now on either the G or so you want to go search by quota name at the top because the first ones are all spot instances. So search by quota name and type on hyphen demand. And these are your choices for that. And then. Yeah, so G is. We think G is going to be a better price performance thing. G will give you better performance, though.

JJeffrey Wagner

I think G's yeah, we want the cheapos.

DDavid Swenson

Yeah, that's what, that'd be my guess. I mean, you're nothing. You don't have a great urgency to get the results from these things. You just want to make sure you run.

EEthan Holz

The other thing is too, is like the base tier of the G's and the base tier of the P. The g tier is like a little more performant at the base and it's cheaper. And the other advantage is you don't have to use, like, remember how I was talking about, like, we had to make our own machine images to like, get our operating system updated on the g instances? Those are all supported. So, like, we don't have to build anything that'll work for the foreseeable future even without us. But yeah, probably ge to start. And then if you end up needing more compute, then P is probably a better option for a lot of cases for you.

DDavid Swenson

Yeah. And then you can request an increase at the account level here. I do notice that you are in us east one, the northern Virginia region, which is convenient because that's where we've already been putting a few things. It's not convenient. Well, I don't know on g instances yet because we're waiting for our quota to increase. But we had trouble getting quotas or trouble getting instances of the p three at all. They were just, they were overcapacity. And since us east one is the kind of default region in a lot of ways, I wonder if another region might have more capacity. But, yeah, we haven't had a chance.

JJeffrey Wagner

To do that yet. I'd be okay to experiment with that. Maybe we try us west.

DDavid Swenson

Yeah. The only thing, a couple of things here. First off, the. So I believe that when you request a quota here, quota increase, it is region specific. And so you'd have to do it again if you change regions. And thing two, is that right now? Well, if you need the image that we're building, which it sounds like if you're using g instances, you won't. But if you need the machine image that we're building that's region specific, we can export it to other regions, but we need to know which ones you're in.

JJeffrey Wagner

Okay.

DDavid Swenson

So it shouldn't be hard for us to make that available in us west one. But it's just like if you don't tell us you're there, we don't do it, and we don't know to.

JJeffrey Wagner

Okay, then let's keep things maximally simple and I'll stay in us east. Okay. On demand g and bt and then request increase.

DDavid Swenson

Click.

JJeffrey Wagner

Yep.

DDavid Swenson

And there'll be a simple little form here. How many do you want? More than zero.

JJeffrey Wagner

Is this basically how many GPU's would I want at peak?

DDavid Swenson

Yes. Yeah. And I would say, you know, expect a smaller number will get approved more quickly.

EEthan Holz

How many?

JJeffrey Wagner

Like ten might be reasonable.

DDavid Swenson

We went up to eight one of ours and on my current request we're at four and I'm still waiting to hear back. I can tell you how the larger.

JJeffrey Wagner

Increase take more time. Okay, so let's do a smaller one then. Let's do four.

DDavid Swenson

You can always increase it more later and most likely what's going to happen is you're going to get, almost immediately, you're going to get a rejection and then you're going to have to explain why you want it. That's just the process. But the first request is easy. You just click request and you're done. As soon as this thing is done, it'll say, yes, you've done this request and then you don't have to think about it. You will get emails at whatever email is associated with this, just so you're aware of that. Yeah, whatever that is. It should come in pretty quickly after. Let's see if we look at the AWS thing again.

EEthan Holz

It'd be funny if they got approved and we didn't.

DDavid Swenson

That would be hilarious.

EEthan Holz

Yeah, we also.

DDavid Swenson

You can also always check the status if we go back to the AWS thing, the actual quota request forms. There we go.

JJeffrey Wagner

Yep.

DDavid Swenson

And so this will have the.

JJeffrey Wagner

Oh, and then it just sends me back.

DDavid Swenson

Yeah, yeah. But you can also do that from, if you go to your quota request history on the left there and it'll eventually. Yeah, you can go to the support case number and you can also interact with it here. This is actually what I've been doing because I don't check the email address that we have. It's only used for admin things. So.

JJeffrey Wagner

Yeah, I.

DDavid Swenson

But, yeah, so anyway, we got that process started. That's not the question you had, but that's the thing that might take a week anyway before you can do anything. Okay, let's get it going to the question that you had. There are. So if you go back to the main dashboard of that, we can just do this from your side or actually let me do it for mine for a second because it's worth showing a couple of things. So there are two different things that are very closely related and very easy to get them confused. One is Iam and the other is Iam. Identity center. What we would recommend is any human accounts that you create go into the IAM identity center, which you'll have to set up. And so you'll see in this it's also region specific.

DDavid Swenson

And so you'll see we have three users here. I have an admin user, then I have two users that are just associated with our email addresses. This is how we would normally use everything with AWS. I actually almost never use the admin user now because I just use root account because I want the admin user can't see the billing information and I want to see how much money we're spending. I use the user account for that. You can set permissions on this, so it's a mess. And honestly, there are tutorials online that will go through it step by step to do this. But we are both using something that's a combination of, there's a built in power user access, there's a built in IAM full access.

DDavid Swenson

We are both using a combination of those two that we did manually, which is, you can just see in here, it's just these two things. So that describes the settings. We have those grouped into a group so that the settings ability or the permission sets are associated with a group name, which we call power users. And then in that group we've added the two of us. So that's kind of the human management. Does that make sense, the kind of groups permission sets? And for the most part, could you.

JJeffrey Wagner

Give me the overview on this and then if you've got time, we could do it for us?

DDavid Swenson

Yeah, okay. That's kind of what I'm trying to do. The only thing I'll say is like, you can get into these permission sets and look into great detail at what they actually are under the hood. It's all a lot of JSON. Let's see. Actually that's. But actually. Oh, here we go back. Yeah, so it's a whole bunch of things like this, but the console makes it so you don't have to think about all this stuff. But if you want to look at it, you can. And it makes sense that it says things like it allows the IM create service linked role within the IM process. It allows list roles within all the organizations. I can understand who are the organizations. And so this is something that just goes on and on from all the different little details that it handles.

DDavid Swenson

But hopefully you will almost never need to look at it at that level of granularity, just to know that it's there if you do need it.

EEthan Holz

David, to note though for the GitHub actions runner, we provide you with that JSON. When you go to create the user you will use for your runner, you can just dump the JSON right in. You just copy it right from our docs, put it right in there and it'll set up with the minimum required permissions to provision new instances on your account. And it has no other permissions other than that. Can't see anything else. You can't see billing or anything like that. All it can do is just create an instance, modify it, and shut it back down. That's all it can do.

DDavid Swenson

That way you can't do anything else.

JJeffrey Wagner

This is interesting. This is almost suggesting that maybe we don't need human accounts at all and we could just ping pong GitHub actions.

DDavid Swenson

I don't know that you do. I'm just saying if you use human accounts, I recommend using IAm identity center.

JJeffrey Wagner

Yep. Okay.

DDavid Swenson

That is what they recommend.

EEthan Holz

They're good to have if you want to play around with stuff. Oops, sorry, David.

DDavid Swenson

Yeah, but as you say, that's distinct from IAM itself. So it's so confusing. Iam, identity center, iam. You'll also see this called SSO single sign on. So it's like Amazon's own single sign on process, which is honestly a much clearer name and less confusing. But now regular Iam, you'll see we have two users right now, one of whom is the terraform test setup stuff. This is what Ethan actually uses when he is doing this stuff for every day. That gives him a little bit more access to try and set things up and try and debug when things aren't going well. And so you probably will not need a user like this. Although if you do, we can give you suggestions on that. You will want a user like this one though, which is the one that actually runs it.

DDavid Swenson

So this is the thing that has the credentials to do this. This is the policy statement. And you can see it allows you to run instances, terminate instances, describe instances, and describe instance status. That's all it can do. And so yeah, it's very minimal in that set. And yeah, we do have for an access key which is going to be needed for it to sort of log in. Is that how it's working right now, Ethan? Use that as a secret. Ethan, you're muted, I think.

JJeffrey Wagner

Yep, yep.

EEthan Holz

Sorry about that. But basically how it works is you can generate an access key. The access key comes with two parts. So you get an access key, which is you can think of as like your username and then I you'll get like a secret access key, which is what your password is. Both of them are used as secrets on our end. So when we set up the GitHub.

DDavid Swenson

Action, you will set your repository as.

EEthan Holz

Secrets in your repository. And then AWS has its own sign in action. And what that does is it sets the environment variables correctly. There's a couple of reasons for that not necessarily needed here, but what it does is it allows us to authenticate it, make sure that all the AWS environment is set up correctly before it runs our stuff. So ours. It basically ensures that any of the things that the SDK might need are set up correctly beforehand. AWS sets that up and then it calls out our runner or our action, which will set up everything for you. But you get one access key and then access key, secret access key, both of those get set in your secrets and then you will use them to log in on the action itself.

DDavid Swenson

One thing we are hoping to do in the future, but not too immediately, it depends on the people. If you end up using this and liking it will probably do this. I don't know if you've done anything lately with releasing to PI, but there you can also do this thing where GitHub is just trusted so you no longer have to give a token. And AwS allows the same thing. It's what is it called?

EEthan Holz

Yeah, it's OIBC, it's open ID connect. So how it works is like it trusts GitHub and then GitHub sends an authenticated request to AWS, it says, or to Pypi, it says this is the repo that I'm authenticating, here's the repo I'm coming from. And it goes great, cool. I know that this was set up intentionally from person who owns this account. And then it can assume that those roles, so like kind of the policy that we set in place, it can assume that and set that up correctly. So we have some stuff in place and some documentation available to make it. So you don't even have to create users, you just create the policy and then the actual action does that automatically. So then that way you don't have to rotate any your access keys.

EEthan Holz

The access keys are only allowed for the time, for like the amount of time that it is part of the runner. So like as it's the step is there and those keys get reset. So they don't have like this where they're like a long lived key. They are like basically one time use and then they're done. And it's a lot more secure, but it also makes it a lot easier for you guys as well. But this kind of has a lessen. It's an easier cognitive understanding. It's easier to talk about, I think, using access keys rather than OIDC.

DDavid Swenson

Yeah, I mean, here you're just doing giving. You're using a secret as a login and a password. And the beauty of the OIDC thing is that you no longer need the login and password. Right. It just. It knows that it's coming from a GitHub repository that you've already said, you know and control. And so, yeah, it allows you to.

EEthan Holz

Not need to use any secrets is what it does is AWS generates the secrets for you on your behalf and then handles it for you. You just have to do some configuration ahead of time so that it knows to provide those secrets when it needs it.

JJeffrey Wagner

Okay.

JJeffrey Wagner

I think I'll stick with the secret based workflow at first until you guys formally recommend to switch over. But thanks for the understanding. Also this. My quota request just got approved already.

DDavid Swenson

Jesus. Hold on. Let me go to. Let me take my support stuff here and see if we've been approved yet. Let's see my support cases. Oh, I must have gotten something back on it pending Amazon action. Yeah, yeah. This is what I had happen last time I had one of these. And it took a week after I got this email before they actually approved it. Man, they must just not like me.

JJeffrey Wagner

They said it may take 30 minutes, up to 30 minutes to apply to propagate.

DDavid Swenson

So I think that in terms of the authentication stuff, I think that answers your questions. Is there anything else we had questions on in this?

JJeffrey Wagner

What is the state of the runners? I guess is there alpha documentation that we can start working with or do you plan to have that out on a later date?

DDavid Swenson

Ethan, can you pull that up and I'll.

EEthan Holz

Yep, I can probably get to it.

DDavid Swenson

Faster than I can.

EEthan Holz

Yes.

DDavid Swenson

We go. I think that we still have a pr open for the most complete documentation. So if you can show that one.

EEthan Holz

Bear that instead. Can you see us a little bit bigger?

JJeffrey Wagner

It's a little small.

JJeffrey Wagner

Okay.

DDavid Swenson

One more.

JJeffrey Wagner

Maybe that twice more.

JJeffrey Wagner

Oh.

EEthan Holz

Yeah. There we go.

JJeffrey Wagner

And let's see, like one more. There you go.

EEthan Holz

Is that okay?

JJeffrey Wagner

Perfect.

JJeffrey Wagner

Yep.

EEthan Holz

So this is the fork of like the docs here. All that's really here is. So we have the breakdown of the setup process here. So this kind of shows what you can do. I have to make one change here to this. Now that I look at it, but basically this is all going to look pretty much the same. The other example we have is open MLM PP test. We have an actual usable example that we're also using internally. This is doing the open MM test. This is the bare minimum you need to get it running. This will all be updated with the. We're going to cut a new version this week so that you can use that instead of using the hash for this. But this is what it looks like.

EEthan Holz

We have this part here, kind of the setup parts, and that there's a lot more here. If you want to add more things, there's more for this. Ideally you shouldn't need a whole ton of it. This is kind of more of a simpler example. But the open mm GPU one, I will include docs on that as well to kind of point to. That's what we use to test. It's something that like we are actively developing on and making sure that always works. So it's kind of a good point to look at. The other piece of docs that we have is we're going to have in this docs repo, we're going to have the list of the different providers we support. So right now it's just aws.

EEthan Holz

So you'll be able to go to aws and it walks you through like how to go through the AWS console, create all the tokens you need. So you'll need a GitHub access token and then an AWS token. Set everything up. Here we have some im or Ami stuff. So those are your machine images on like what to do here. Some GPU instance recommendations for you want to use P two instances or other instance types. What kind of what we recommend here. And like what we recommend is useful. Like on those different machine images. Yeah, this is kind of just a list of resources. So if you're interested in like setting up the account, this is exactly the document you'll use. It kind of walks you through that whole process. And then the Ami side is kind of where things get a little hairy.

EEthan Holz

But feel free to let us know and we can definitely adjust that as needed. As for additions, like I said, these in the chat. Yeah, so for right now, it's here. Where is the chat?

DDavid Swenson

Yeah, we should link to the two actual repositories as opposed to your four.

JJeffrey Wagner

Yeah, there we go.

DDavid Swenson

At the end of what you're seeing is from a branch that Ethan is doing right now then. Yeah, here's the branch.

EEthan Holz

So here's the actual runner. This is the PR with the docs.

JJeffrey Wagner

In it.

EEthan Holz

So keep an eye on when those get purged and any changes we make there as well as any discussions we're having on that. Feel free to keep an eye on those if you're interested. Then the other one is. Here we go. This as well, which is the test bed that we're using for actually using this thing and using it to test it. Test a couple of things. We test docker like Nvidia installation is correct. And then also to make sure that Openo is able to see that installed Nvidia installation. So we have all three parts here.

JJeffrey Wagner

Okay, just taking I did start a note stock and so I'll invite you to add the Fireflies notes as well. Okay. Do you guys consider this ready to use? Like if we start mucking around in this and asking you questions, will you say aha, thank you for helping us polish or oh no, we knew that was wrong and now you're just annoying us. Which phase is this in?

DDavid Swenson

I think it's a lot closer to the helping us polish. I mean, you know, we haven't really tried it on anything that we're testing, but the fact that we have this open, open mm GPU test. So that's enough that we are sure that it is seeing Cuda on the device. We know it works on a p two. Obviously we don't know that it worked on the g instances yet because you got you approved before I did. But as soon as we get that approval, we'll check that in principle it should work. It's just a matter of making sure that does with an AMI that it's reasonable.

EEthan Holz

The only thing I will say is that the docs will go through a little bit more change through the end of the week. So our goal for this is to have a, like I said, cut our release this Friday. David's like out of town on Friday. So like we're going to have that done by then just to kind of clean up a couple of docs things like I said, include links to our OpenMm GPU test, showcase that as and kind of make our examples more close to that, then do more complete examples so that they're more useful, I would say, and get that all kind of polished up by Friday. So my recommendation would be that if you like, get to Friday or Monday of next week, that would probably be a better time.

EEthan Holz

But feel free to dig in right now if you want to. But that's kind of where we're at on usability in terms of documentation.

DDavid Swenson

I'd say life will better if you wait until Monday. But that's, you know, we're very close to that. I mean, at that point, we'll have a proper release.

JJeffrey Wagner

I'm glad I could ambush you all with a fast AWS approval. I knew that flipping a 20 into their Venmo account would help, apparently.

DDavid Swenson

Yeah.

JJeffrey Wagner

Okay, then these are good resources, and we can easily wait till Monday. But I'm glad that we have notes on this. And I think looking at this, the hello world, the openmm gpu test. Yeah, this guy. Fantastic. I can look at this, and I can understand it. So then the big question mark is account setup, which I think now I'm in a technical. We're technically able to do account setup, and I'd be happy to try doing that now in case there's, you know. And I could do it while reading your documentation, and you could see me stumbling on stuff, and maybe that would help, or what have you. Or if you want to call it a day and tell me to read the docs, and that's fine. Cause you guys actually.

EEthan Holz

No, no. I'm actually okay with, like, us kind of going through. If you want to use the docs that we have available, feel free. And then that way we can kind of see where the pitfalls are. I'd really like to see what hangs you up, if that's okay with you, just so that I can make those changes as we need to.

JJeffrey Wagner

Absolutely. David, does that sound good to you?

DDavid Swenson

Yeah, yeah. The one thing I did also want to say about all of this is that depending on what your needs are, I mean, this is very much. The fact that we're also integrating this with GitHub actions is specific to the idea that's part of what you need. And if that's not how you want to run stuff, there might be easier ways to get things set up, if that makes sense. I mean, it's kind of. If you just run on Aws directly, if that's a better solution, we do.

JJeffrey Wagner

Want to run on GitHub actions.

DDavid Swenson

Okay.

JJeffrey Wagner

Yeah, 100%. That's the purpose. Okay. And I want to ask one clarifying question. So, there was IAM identity management, and there was IaM. And as best practice, you recommended IAM identity management for humans and IAM Identity center.

DDavid Swenson

Identity center.

JJeffrey Wagner

Oh, identity center.

DDavid Swenson

Yeah, yeah.

JJeffrey Wagner

And then Ethan mentioned that there was a bot user that he sometimes used for interactive sessions, I believe. Is that correct?

EEthan Holz

I'm using it on terraform, which is the CLI tool. I'm using it from my CLI to just do one off provisioning requests, it would be the equivalent of what I was doing in the action. But I'm running it locally. It's more useful for me testing the things that I want to test locally without having to. It's a similar secret set to what I would have actually on the action itself. It's the same idea, but it's not like actually me using it on the actual portal. It doesn't have a way to log in. You can't actually log in with that user. It's only used to authenticate to create resources.

DDavid Swenson

It's when.

JJeffrey Wagner

Summary of what terraform is.

JJeffrey Wagner

Yeah.

EEthan Holz

So terraform is an infrastructure as code platform. So it allows you to write a declarative document that says, I want to provision this instance with these parameters and I want it to do this. And then it saves the state of that so that you can, when you're done with it, you can tear it back down in the same way that you put it up. That's kind of like what terraform does.

JJeffrey Wagner

Okay. Yeah.

EEthan Holz

It's kind of out of scope for this.

JJeffrey Wagner

Yeah, sure. I'm trying to distinguish. So basically, option a is the very simple option where I just make a bot user for RCI. And when we need to run stuff and debug stuff, we just have to keep pushing changes to a pr on GitHub. And I guess the old. And so that might become burdensome. That would annoy me. It'll probably annoy Matt. And so should we be. Is there a less painful route via making personal accounts or having like a manually available bot account?

DDavid Swenson

So keep in mind that what Ethan needs to debug here is the code that is our GhA runners code. Ideally, if you've got a problem with that, we'd love to have help if you want to. But also that's where you just create an issue and say, guys, your code's broken. If you're working on that, you might want to be able to provision your own AWS resources, right? If you're not working on that, then that shouldn't be what you need. What you could want, though, is to be able to log into like a running action. We haven't tested whether that's even possible. There's a. I don't know if you've ever tried that. There is this. There's an action that allows you to do that you can actually put something in your GitHub actions as a step in your GitHub actions.

DDavid Swenson

And it will freeze there and give you an ssh thing. We haven't checked whether that would work on this. I tend to think it might not because we've probably locked down the ports pretty well by default.

EEthan Holz

By default, the security group will just.

JJeffrey Wagner

Let you.

EEthan Holz

You should be able to, unless you change the security settings yourself.

DDavid Swenson

But by default, yeah, we all play with that one and see if it works. But that would be, I mean, it's already something that you can, that's what you have to do that on GitHub, right. If you wanted to check something out while your action was running, and you want to see if it's specific to the hardware that it was on GitHub, you need some way to ssh into that box and there is a tool that allows you to do that. I still don't understand how it works. It seems like magic to me, but I like it.

JJeffrey Wagner

Okay, so let's assume, yeah, that's a good point. Like let's assume that we're setting this up for non interactivity first. I think you guys have made a great thing. I just worry we're going to have obscure errors that are ultimately due to, you know, some particular CUDA firmware or virtualization of CUDA firmware or something. And we will want to be able to investigate it a little bit before we just write to you and yell that it's broken, because it could easily be stuff on our side or on the contour forge side and.

JJeffrey Wagner

Yeah, okay.

EEthan Holz

There is some concern. So a couple, one thing to note. So when you're using G instances, they have a pretty comprehensive operating system set up that basically like Ubuntu with CuDA 12.4 and then some other fun, make sure that all of that is done ahead of time. If you run into any of those issues, we want to know about it regardless, because it's something that we want to document internally. We want to make sure that if there's any of those errors, that those are documented, because CUDA errors just seem to be, everyone has the same issues, but a place where all of them are documented doesn't seem to exist. So having that there is really helpful to us as well. So feel free to open issues if there is nothing. And then we can help make that environment more reproducible for all of you.

EEthan Holz

But also it helps for us to document for people in the future as well.

JJeffrey Wagner

Okay, cool.

JJeffrey Wagner

Yeah, we can do that. And yeah, we'll give you lots of feedback and tag you insistently on GitHub.

DDavid Swenson

Please do be useful.

JJeffrey Wagner

Okay, well, neat. So then I'm going to try setting up an account. Matt, do you have any thoughts as to. Do you want. Do you want a personal AWS compute account to do experiments? Or would you be happy with the bot account, Matt, would there be restrictions on a bot account? The bot account.

DDavid Swenson

The default bot account, is extremely limited. The thing that we recommend using as what the actual thing that talks from GitHub actions to AWS is extremely limited because we don't want it to do anything other than the things it needs to do for its job.

JJeffrey Wagner

Okay, so let's get Matt and I full accounts just in case, and then we'll also make a bot account, and that's what we'll plan to hook up to the GitHub actions. So let's start with the human accounts.

DDavid Swenson

That's identity center.

JJeffrey Wagner

I assume.

JJeffrey Wagner

I want to enable this.

JJeffrey Wagner

Yep.

DDavid Swenson

Note that it is specified to a region, but you can only have it for one region. So it's kind of weird like that. It's both. I don't understand why it's that way. It is like a global thing, but somehow it has to be specified within a region.

JJeffrey Wagner

That's weird.

JJeffrey Wagner

Okay.

DDavid Swenson

Yeah, so your account can. Your account now has IAm identity center in us east one.

JJeffrey Wagner

Us east.

DDavid Swenson

You could still use it in principle for the work you were doing in us west one, but the login is still associated with us east one. It's. I. I found this really confusing. And you can't set up a separate one within your account in us west two because west one. Because that would just be weird to have two different login processes for yourself. Anyway. So from there you can go ahead and start creating a user. Go to the users on the left. Add a user standard information.

JJeffrey Wagner

Yeah.

DDavid Swenson

We've been using our email addresses also as our usernames, but it doesn't matter. So the AWS console also has this thing that feels kind of like a dependency inversion sometimes. So, yeah, you don't need to add all the contact info. It's got all that stuff but keeps not scrolling.

JJeffrey Wagner

Okay. Yeah.

DDavid Swenson

And so now you can create a group. This is part that feels like it's a weird inversion thing. And then, yeah, you can create the group. Let's see, where was it? How does this work? I think. Did it pop up a new thing? Go back one. Yeah. Now hit the refresh thing at the next. Create group. Yep. There you go. Now you've got developers. This is this really weird AWS workflow console workflow.

JJeffrey Wagner

Yeah, Matt, any typos? That all looks good.

DDavid Swenson

Let's see. Add the user. And now is the confusing part, which is adding. Yeah, so, Matt, you'll get all that stuff and you'll have to do that thing. You don't just go to Aws, Amazon to log in with this. You'll have a special URL that is for your account, associated with your account, but that's also easy enough to deal with. I think you may also have to turn on. I'm trying to remember how this works because I've had to do this a couple of times, but I'm looking through my own just to make sure I can figure out how that works.

JJeffrey Wagner

Okay. Yeah, tell me to register. MFA.

DDavid Swenson

Yeah, that is definitely recommended.

JJeffrey Wagner

Okay, Matt, have you successfully done multifactor? I'm logged into a console. Anything other than that, I have no idea.

DDavid Swenson

All right, so right now you may need to get back to the. Jeff, you need to get back to the root account to because you haven't finished giving yourself permissions yet.

JJeffrey Wagner

Oh, okay. I wonder if I can be both at the same time.

DDavid Swenson

Only if you're lucky. Let's see what happens when you try and reload. Good. And this is part I have to try and remember. Yeah. This is where you have to turn on AWS organizations. I believe in order to be able to do permissions from here, you see related consoles, AWS organizations. This is the part that got me confused when I had to do this again. Okay. Do we not have. This is weird because it's under OMSF, click under policy management here.

JJeffrey Wagner

Oh, that is weird.

DDavid Swenson

Yeah, I think that this is yours is different from mine. Okay. Which means you can't see your billing information because you're not in charge of it. Okay. I'm going to try to figure this out for a second.

JJeffrey Wagner

Oh, interesting. So I'm logged in as info at OpenFf, but my management account email address is info msf.

DDavid Swenson

Yeah. So AWS has this nested account system thing. And so that can be nice in principle, because that means that the bills are going to Carmen instead of you. But I don't that changes some of how this works and on ways I don't know, because you're not like the true root user over everything. You're just root within this account. What I don't understand, I'm surprised that the thing I thought I wanted to do here isn't possible. If we get a.

JJeffrey Wagner

I can add AWS accounts. Okay, so this might be a Carmen thing.

DDavid Swenson

Yeah.

JJeffrey Wagner

Okay.

JJeffrey Wagner

This could be a good ending that we have reached this point and we.

DDavid Swenson

Are confidently confused and yeah, as often happens with AWS.

JJeffrey Wagner

We need Karman. Yeah, no, that's fine. I assume that OpenFe and other OMSF projects will have similar setups. So good that we ran into this.

DDavid Swenson

Yeah. If you want to work around for now, what we can do is do what's not the recommended best practice. So the best practice is to do all of this stuff within IAM identity center. But you can also just create a regular IAM account for Matt so you can actually use it.

JJeffrey Wagner

Oh, because we're still in. Oh yeah.

DDavid Swenson

Again.

JJeffrey Wagner

But we can go back to vanilla Iam.

DDavid Swenson

Yeah. There, it's a little bit more limited because you're basically, you're very much stuck with this user path type of approach. And so it looks like you have two users now already. What are the two? Probably.

JJeffrey Wagner

Yes.

DDavid Swenson

David Dodson had an account some two years ago. So yeah, you can just create a user here.

JJeffrey Wagner

Okay. And so this will be for Matnight. This will be a new, this is uncoupled from the previous account.

DDavid Swenson

Right. I personally recommend giving it a slightly different name because let's just make the.

JJeffrey Wagner

Bot user right now.

DDavid Swenson

Okay.

JJeffrey Wagner

Yep.

JJeffrey Wagner

We'll do that. And then we can.

DDavid Swenson

Why don't we.

EEthan Holz

Oh, we'll do that in this here. So just name it whatever you want. You can, you can pull up the docs. Yep.

JJeffrey Wagner

Perfect.

DDavid Swenson

It does not need access to the management console.

EEthan Holz

So, no, that means it can't log in. So that's kind of sweet. And then you can do, so what I've done is you can do a task policies directly. That's what I've used. If you wanted to create multiple users for like maybe you wanted every single instance to have different users. So you can see which users are using more compute. You could set this up so that you create a policy and attach it to a group. Then all of them will get the same. But we'll go here and then you can select create policy on the top right of the permissions policy right there. What I recommend here is I use the JSON editor because I just copy.

DDavid Swenson

Paste what we have.

JJeffrey Wagner

Yeah. Open. I did have this pr.

DDavid Swenson

Open.

EEthan Holz

It should be the, you go.

JJeffrey Wagner

Is.

EEthan Holz

That, okay, that's the base one. So if you go, oh, wait, is.

JJeffrey Wagner

That, no, yeah, this is base point, your branch.

EEthan Holz

Okay, that's fine. So if you go down to docs, aws.

JJeffrey Wagner

Yep.

EEthan Holz

So you scroll down, there's that JSON.

DDavid Swenson

You should just take the copy button in the corner.

EEthan Holz

And then you'll just replace.

JJeffrey Wagner

I can find it.

DDavid Swenson

Yeah, there it is.

EEthan Holz

Just replace it.

JJeffrey Wagner

Then there you go somewhere at the.

DDavid Swenson

Bottom there should be, oh, what do you need the arn?

JJeffrey Wagner

Yep. Let's see.

EEthan Holz

Have it up here. See this is why we do it live. Score. When, let me check my user.

JJeffrey Wagner

Permissions.

EEthan Holz

This one. Oh, resource should be star. So what that is.

DDavid Swenson

Yeah, I think it should be star, right?

JJeffrey Wagner

Yep, yep.

EEthan Holz

So inside the quotes.

JJeffrey Wagner

Yep, right there. Then there you go.

EEthan Holz

And then you just hit next.

JJeffrey Wagner

Yep.

DDavid Swenson

And then give it a name probably, you know, gha runner policy.

JJeffrey Wagner

Cool.

EEthan Holz

So then you can go back up to where were before it did.

DDavid Swenson

The thing where it popped out a new tab. So you gotta go back one and refresh.

JJeffrey Wagner

Refresh.

DDavid Swenson

If you go filter by type, customer managed. There you go. So this is attaching a positions policy directly to that user. Now to actually use it, you'll have to go to that user and create access keys. So you see where it says at the top access key one create access keys link. Oh yeah, to do that, yeah, it tries to tell you not to ever do this, which is what all of this is for. So this is really like you surely shouldn't do this. You shouldn't create a user and a password here, but you're going to anyway.

DDavid Swenson

So yeah, you can click one of the options and then say, yes, I really want to do this anyway, or you can click on other and it'll be like, okay, well maybe click on any one of these and they'll be like you have this other option, you know.

EEthan Holz

Yeah. You click on something and it'll say hey, here's some great docs on how not to do this.

JJeffrey Wagner

Yeah, yeah.

JJeffrey Wagner

Okay, sure.

DDavid Swenson

Which is both utterly confusing and actually really useful. I think it'll show us not the key, it doesn't show the key, but it shows the key, but not the secret access key. The access key is your username, the secret access key is your password.

EEthan Holz

You can also just download the CSV as well and it'll download it for you on the bottom and it'll have.

JJeffrey Wagner

Both for you if you want to.

DDavid Swenson

I recommend that especially doing it live. You don't have to even hide it from us.

JJeffrey Wagner

Yep.

EEthan Holz

Okay. And then you're done. That's pretty much it.

DDavid Swenson

So from there you would need to put the access key and secret access key into your secrets in the repository that you want to test. And you also have to create a GitHub token as well because it has to go talk to GitHub programmatically as well as AWS programmatically.

JJeffrey Wagner

So if you don't mind, do you want to give it a shot? I could open a pr two evaluator that just runs our hello world.

DDavid Swenson

Yeah, there's more stuff to set up along the way, especially around, like there's that whole, there's a reason there's a very long step of do this, do this. Because you have to tell the, again, you have to get your personal access token for GitHub as well and then add everything into the secrets. Of course. I think we're good on the AMi and then it should work. Ethan, isn't there something in here that you have to set up for self hosted runners in the repository settings?

EEthan Holz

No, you don't have to do anything there. It'll just request from GitHub itself. Like GitHub will provide you with a token. So like our tool asks GitHub for the token gets turned back. We use the token to then like re authenticate with GitHub to say here's a runner. And then what it'll do is you can just set it to say like I want to run on a self hosted runner and it'll run self hosted.

DDavid Swenson

This is if we want to do runner, different groups is where you have to do some settings if you want to have runner groups. Right?

EEthan Holz

Yeah. And then the other thing is, so.

DDavid Swenson

Some things you want to run on self hosted gpu, some things you want to run on self hosted CPU, you have the ability to put labels on it that will do that, but that's more advanced to set up, you.

EEthan Holz

Get a temporary label. But.

DDavid Swenson

Jeff, when do you have your next meeting with Carmen, or when would you be doing that? Okay, because I'm just saying we have one, and this has come up a few times in the past. We have one a week from today, except for it's probably going to be a day earlier because we're supposed to be changing the date on that. But so we might actually bring that up as well because he's wanted us to bring our root account under the OMSF umbrella as well, which makes sense and would mean that instead of being on my ramp card, it was on, you know, some official funds somewhere. But yeah, I'll mention this here, because it's the same problem. And we won't be able to do it until she does that, I think either.

JJeffrey Wagner

Okay. Yeah. Do you want to add me to that meeting?

DDavid Swenson

I can, yeah. It's part of our standard, it's just a standing meeting with the three of us. So there'll be a bunch of other stuff there that you're not going to care about, but you're welcome to come to the part you do care about if you want to, up to you whether you want to. We can try and order the agenda so that we're not wasting your time, at least with the stuff you don't care about.

JJeffrey Wagner

Here's the problem. If I schedule my own meeting with Carmen, I'm going to say, carmen, could you help me fix these user permissions? And she'll say, how do I do that? And I'll be like, I don't know, I need to talk to David.

DDavid Swenson

Yeah.

JJeffrey Wagner

And so we'll need to all be in the same place. The question is do we want to hijack your meeting or.

DDavid Swenson

To a large extent I think that, I mean to, I think what happens is she needs to log in and click that she needs to get another email for that account. Yeah. To be honest, you could probably just do that. Wherever was the send verification email, you could probably just send that again and see if she gets it and handles it.

EEthan Holz

David, this might be something we want to meet with her on anyways and just try and figure out what's going on here because we'll probably have to do this with at least two other organizations.

JJeffrey Wagner

So.

DDavid Swenson

Yeah. And if we do it just for ourselves first, then that should get it. Because I think it's just a link. It looks like it's a link in an email that she needed to click. And so she needs to send another one.

JJeffrey Wagner

Yeah, let's do it with her just so she knows what she's clicking on. could you decide me to your meeting next week and I'll assume that I'm the lowest priority item on the agenda?

DDavid Swenson

No, no, to the contrary. We'll, we'll try and start with you so that you can, you know, leave and do something productive with your time afterwards.

JJeffrey Wagner

I don't do productive things. Matt knows that. I just have meetings to schedule.

DDavid Swenson

More meetings.

JJeffrey Wagner

Okay.

JJeffrey Wagner

But I think Matt and I have the tools to set up the GitHub action spot. And we can give you feedback if we have any trouble with that.

DDavid Swenson

Yeah. Barring something missing from our instructions there, which is the feedback that we need, if there is something missing, the only.

EEthan Holz

Thing I will say is I think up top where we created the user before, you might have to hit done so it actually saves it.

DDavid Swenson

Did we have one where we hadn't.

EEthan Holz

Quite, I think it's right there. So I think you just hit done.

JJeffrey Wagner

That's interesting. It would let me get the access keys without being done.

EEthan Holz

I think it'll give it to you, but then they're not active until you hit.

JJeffrey Wagner

Done.

EEthan Holz

So they're provisioned but not usable. Same thing with GitHub actions. So admittedly, when we provision your self hosted runner, one thing that we do is if it fails, it will get a token. If the token never gets used, then it's not useful and it expires within 30 minutes anyway. If it just fails and fails, and then we don't care.

JJeffrey Wagner

Okay, cool.

JJeffrey Wagner

I will, yeah, I will work with Matt to advance this.

DDavid Swenson

Yeah. Especially if there's anything unclear in the instructions on setup. If there's anything like that, do just reach out to us immediately because if it's not clear, then that's on us to improve.

JJeffrey Wagner

Okay, cool.

JJeffrey Wagner

All right, I think that's good. We have a science meeting that maybe we should get to, but yeah. Anything else that we want touch on today?

DDavid Swenson

I think that's it.

JJeffrey Wagner

Yeah.

DDavid Swenson

Let me know if there's anything that you, else you need for me.

JJeffrey Wagner

Okay. David, when you get the fireflies notes, could you put them in here?

DDavid Swenson

How do you usually do that? Do you just want a copy of. You should actually get a note as well, right? If you don't, let me know, but I can. Do you just copy the, what is, do you put a link to it or what do you do?

JJeffrey Wagner

Just the raw text.

DDavid Swenson

Okay. Raw text of the transcript or.

JJeffrey Wagner

Yes.

DDavid Swenson

Okay. But yeah, you should have a link to it if you want to as well.

JJeffrey Wagner

Yeah. Okay.

JJeffrey Wagner

Yeah, if I can get access to it, then I'll just do it myself.

DDavid Swenson

I think it'll email you because it's got your name on it anyway.

JJeffrey Wagner

Okay, perfect. Thanks everyone.

JJeffrey Wagner

The recording has stopped.