A[19:06:32] Janet Rossini: are we still sharing servers with other regions? does their script use count on the stats i see? in terms of script time avail?
A[19:07:10] Simon Linden: Yes there are more than one region running on a given piece of server hardware
A[19:07:22] Simon Linden: but their script usage shouldn't affect your numbers
A[19:07:43] Janet Rossini: so each region gets a standard slice of script time?
A[19:08:01] Janet Rossini: what can be said about how long it takes to process a script that doesn't want time ...
A[19:08:16] Simon Linden: you can get situations where the whole server performs poorly if you get an unlucky combination of busy regions together
A[19:08:17] Janet Rossini: does 5000 seem like too many, with most idle or waiting for a chat etc?
A[19:08:25] animats Resident: Someone here want to go visit her sim and help her out?
A[19:08:34] Simon Linden: if you do a restart, that typically moves you to another server and it changes
A[19:08:48] Janet Rossini: that could be worth asking for j ust to see what happens
A[19:08:53] Simon Linden: 5000 is a lot but not crazy-big
A[19:09:05] Janet Rossini: there's no way to get top scripts or the like in mainland?
A[19:09:21] Rex Cronon: 5k running scripts in a sim is a lot?
A[19:09:22] Simon Linden: ctrl-shift-1 shows you the most info
A[19:09:53] Janet Rossini: yes i use that and brought snapshots. which reminds me of two things ... does the HTTP change effect scripts in any way ...
A[19:09:55] Simon Linden: 5000 of anything is a lot, if you have to deal with each one individually
A[19:10:25] Simon Linden: also consider that "scripts run" stat - if the region is running full speed 45 fps, that's about 23 milliseconds per frame
A[19:10:54] Janet Rossini: yes. as i say, most are idel
A[19:11:02] Janet Rossini: idle
A[19:11:16] Janet Rossini: and we stopped all the trains and elephants and moving stuff
A[19:11:35] animats Resident: Can support give her a top-scripts list?
A[19:11:54] Janet Rossini: i'd commit a felony for the estate tools :)
A[19:11:58] Rex Cronon: but there r people with thousands running scripts in their attachments...
A[19:12:54] Simon Linden: so scanning and possibly running 5000 scripts in the time used for scripts is a fair amount of work ... that's all I'm saying. A lot depends on what the scripts are doing too
A[19:13:01] Janet Rossini: anything that would help us triage would be useful
A[19:13:13] Janet Rossini: most of them will be idle waiting for commands on weird channels ...
A[19:13:19] Janet Rossini: or one-time things like texture animators
A[19:13:28] Janet Rossini: except for the trains (which we halted)
A[19:13:52] Janet Rossini: but every script gets assessed for wanting to go every time around, right?
A[19:13:58] Simon Linden: right, or waiting for a touch or collision or whatever event
A[19:14:07] Janet Rossini: right
A[19:14:36] Qie Niangao: I have an alt staring at the sim stats there now... for an idle sim, there's kind of a lot of Physics going on... like about 3ms per frame. Not horrid, but with everything else it does cap Script time to 15-16 msec
A[19:14:37] Simon Linden: Right, it tries every frame to check each and every running script, and it can be in a few simple states
A[19:14:46] Simon Linden: first, it might be totally idle waiting for an event
A[19:15:09] Janet Rossini: offhand i'm not even sure how to search and find all the scripted objects to stop them, they're all over :)
A[19:15:15] Simon Linden: it must update timers and see if that causes an event
A[19:15:30] Simon Linden: or it's in the middle of an event already, running the program
A[19:15:44] Simon Linden: in that case, it gets a short slice of time to run
A[19:15:53] Qie Niangao: World / Show / Beacons / Scripted is a start
A[19:16:05] Janet Rossini: qie i was seeing 2ms physics last night
A[19:16:14] Janet Rossini: yes all 1300 of them :)
A[19:16:21] Janet Rossini: but it may come to that
A[19:16:48] Rex Cronon: there r scripted tools that allow u to find running scripts
A[19:16:59] Janet Rossini: well i don't want to take all the time but ping me with ideas or tools etc please
A[19:17:02] Qie Niangao: yeah, it's not all due to physics; even if that came down to 0 and scripts got the more common 18ms, there still wouldn't be enough time for all the scripts to run
A[19:17:14] Rex Cronon: i mean objects with running scripts in them
A[19:17:17] Janet Rossini: physics and scripts share time tho?
A[19:17:48] Janet Rossini: i can script necessary tools too but idk of much in the library that's helpful
A[19:17:54] animats Resident: Yes, unfortunately. It's single thread.
A[19:17:55] Simon Linden: the 23 millisecond time for a frame gets split up with all the work running the world
A[19:18:17] Simon Linden: so yes, physics gets time, and that one is tough to limit compared to scripts
A[19:18:22] Elettra Beardmore: hallo to all
A[19:18:23] Janet Rossini: anyway we can go to next question and please folks IM me with ideas.
A[19:18:33] Rex Cronon: ehllo elettra
A[19:18:36] Rex Cronon: hello*
A[19:18:53] Simon Linden: it has to figure out all the updates to send to everyone ... checking individually about where your camera is, draw distance, etc
A[19:19:05] animats Resident: The sailors are unhappy with the retransmission time fix.
A[19:19:23] Simon Linden: actually send and receive all the network traffic and other jobs
A[19:19:52] Simon Linden: so yes scripts are sharing time with a lot of other things
A[19:19:58] Janet Rossini: oh i should say we don't think we've changed anything significant. but one never does :)
A[19:21:07] Janet Rossini: yes all programmers should have a tattoo of that :)
A[19:21:15] Kennylex Luckless: Is there any good tool we users can use to identify what may cause lag in a sim?
A[19:21:27] Lou Netizen: /me hides he "It works on my machine" tattoo
A[19:21:39] Simon Linden: the best available info is what you see with ctrl-shift-1, that stats
A[19:21:48] Kennylex Luckless: We have problems in Natoma with lag that often is very hard to identify.
A[19:22:08] Rex Cronon: u need to first stop rezzers
A[19:22:19] Simon Linden: it's often a puzzle to figure out what's doing what in SL :P
A[19:22:29] Rex Cronon: then either stop of push out of sim all physical objects
A[19:22:45] Janet Rossini: yes. we had the bird flocks etc turned off as well
A[19:22:51] Janet Rossini: and the deer who run around
A[19:22:58] Kennylex Luckless: We even try to blame Ebbe for the lag, but that did not help :)
A[19:23:15] animats Resident: It took two months for support to find out the cause of a lag problem in Vallone, where very little was happening. Some landlord had a "land board" that was scanning for visitors, inefficiently..
A[19:23:28] Simon Linden: yeah Kenny I don't think it would go well if I tried that either
A[19:23:31] Rex Cronon: than try and find all objects with a huge land impact
A[19:24:37] Rex Cronon: scan all sim visitors for how many scripts they have in their attachments and how much memory they r using
A[19:25:15] Kennylex Luckless: The problem is that Natoma once was copied to the Beta grid, it was an exact copy, but there was no lag in the Beta ver of Natoma.
A[19:25:23] Simon Linden: that's certainly one way things will change and get laggy ... each AV adds more work to the server and your viewer
A[19:25:41] Kennylex Luckless: So it may not be the prims or script in Natoma but something eles.
A[19:25:46] Kennylex Luckless: else
A[19:25:55] Janet Rossini: it's happening with just us in the region. yes i'm familiar with the impact of avatars, i'm security person at NCI
A[19:26:01] Simon Linden: Right Kennylex - often it's the AVs and region together making problems
A[19:26:02] Janet Rossini: so we see 20 and 30 people much of the day
A[19:26:22] Jenna Felton: could be interesting when the region could keep statistics about what LSL function was called and how often. So you could find out if ancient AOs were in use or there are HTTP requests going permanently or alike
A[19:26:23] Simon Linden: that's a fair amount of traffic, Janet
A[19:26:34] Janet Rossini: yes NCI is a newbie center so quite busy
A[19:26:44] Rex Cronon: ask them to come in naked:)
A[19:26:53] Janet Rossini: oddly we have a rule against that :)
A[19:26:58] Janet Rossini: being g rated and all
A[19:27:13] Simon Linden: it's possible to do that, Jenna, with a special test that takes time to set up. It also runs really slow with all the extra tracking work
A[19:27:26] Jenna Felton: ah, ok
A[19:27:38] Janet Rossini: is there a request we could make to support to help us sort things?
A[19:27:43] Janet Rossini: if it ocmes to that?
A[19:27:50] animats Resident: That's important. Most newbies give up quickly on SL. So fixing this problem will help with customer retention. Help her.
A[19:28:03] Janet Rossini: no no nci is fine this is my region that's faltering
A[19:28:11] Janet Rossini: much as i'd love to have help
A[19:28:11] animats Resident: OK.
A[19:28:18] Janet Rossini: i'm pretty sure if i had god power that would help
A[19:28:31] Janet Rossini: so if you could put a word in ...
A[19:28:33] Simon Linden: but looking at this from a more general level, I totally believe it would be nice if we all could tell what was slowing things down easier
A[19:28:37] Rex Cronon: usually lindens r good at finding problems objects in a sim:)
A[19:28:54] Janet Rossini: is there no way to enable estate tools, top scripts etc in mainland?
A[19:28:59] Simon Linden: that goes for both the viewer and regions
A[19:29:56] Simon Linden: I'm guessing the mainland tools are all in an odd state for privacy reasons - not letting others poke into what you have on your parcel
A[19:30:13] Janet Rossini: srsly tho if there are other questions do jump in
A[19:30:23] Janet Rossini: yes i imagine that's the thinking, simon
O[19:30:34] Meeter/77e48a07-06f1-4ae4-9d33-6eab4d445e2d: Timecheck : User Group is half over
A[19:30:42] Nalates Urriah: What is happening with EEP?
A[19:31:20] Lou Netizen: well, I had a brief update on the outbound HTTP request issues we ran into were it appeared certain server IPs couldn't reach particular external resources. We believe the issues we encountered were almost certainly at Amazon AWS and not within the LL network.
A[19:31:45] Janet Rossini: :)
A[19:32:12] Lou Netizen: When we were able to identify an IP experiencing the problem, sims on that IP were able to access resources at other providers normally.
A[19:32:13] Simon Linden: and I didn't answer - yes Jenna please file a support ticket about the performance being poor there. I don't know if you will actually get help, because the staff has to rank the crisis and panic levels all the time and that's a tough one to figure out
A[19:33:05] animats Resident: Re HTTP requests: what HTTP error code?
A[19:33:09] Lou Netizen: 499
A[19:34:06] Rex Cronon: AFK...............................
A[19:34:07] Rex Cronon: afk
A[19:34:07] animats Resident: 499 is SL side.
A[19:34:27] animats Resident: Sim out of resources or hit rate limit.
A[19:34:58] Lou Netizen: yep, which is why I thought it might be an SL issue. And maybe it is. But it's not rate limiting, and the error only occurred on particular sims with script attempting to access AWS resources.
A[19:35:17] Lou Netizen: same sim, same IP, differnet external resources…all fine
A[19:35:22] Simon Linden: yes, that error code comes from Linden servers, where basically something is hosed and it can't proceed
A[19:35:37] animats Resident: Docs also say that no response in 60 secs gets a 499. Or an SSL error.
A[19:35:47] Lou Netizen: yeah, these 499s were immediate
A[19:35:48] Simon Linden: it may be due to some external problem, however
A[19:36:04] Janet Rossini: so it could mean AWS said something they couldn't understand
A[19:36:12] Lou Netizen: my working theory is some security appliance at Amazon is a little overactive
A[19:36:21] Janet Rossini: could it be related to the newish requirements for HTTPS to work?
A[19:36:44] Lou Netizen: Not so far as we can determine, Janet. All these tests are HTTPS.
A[19:36:53] Simon Linden: sure, it could be anything in the network layers
A[19:37:22] Lou Netizen: Yep, but when it only happens with AWS resoruces and not to Azure, Linode, private servers, etc…it's probably Amazon
A[19:37:30] Janet Rossini: /me nods
A[19:37:46] Lou Netizen: the trick was just finding IPs on the grid experiencing the problem so we could test
A[19:38:00] Lou Netizen: and the problems do seem to clear eventually
A[19:38:05] animats Resident: Is it your service on AWS? Do you control the other end?
A[19:38:07] Lou Netizen: some IPs that experienced the problem a week ago are fine now
A[19:38:19] Lou Netizen: Yes, the service where we ran into this problem is AWS.
A[19:38:37] Lou Netizen: I do not personally control the other end; I set up a free micro instance for testing though.
A[19:38:39] animats Resident: Can you look at logs from that end?
A[19:38:57] Lou Netizen: Yeah, in the testing, Amazon's server does not log a request
A[19:39:02] Lou Netizen: so it never gets there
A[19:39:08] animats Resident: OK.
A[19:39:22] Simon Linden: did you hit that instance via an ip address or domain name?
A[19:39:28] Janet Rossini: dns thing?
A[19:39:35] Lou Netizen: for the "live" service via a domain name
A[19:39:42] Lou Netizen: for my test scripts, via an IP
A[19:39:51] Simon Linden: ok yeah so dns has to work right on that
A[19:40:02] Janet Rossini: it fails on your tests lou?
A[19:40:05] Jenna Felton: can the external servers bloch access from SL like the mail providers drop the mails from SL servers as spam ignoring approach?
A[19:40:06] Lou Netizen: results were consistent across both: if the live service on AWS wasn't reachable, neither was the IP
A[19:40:07] animats Resident: What DNS does SL use?
A[19:40:09] Jenna Felton: block*
A[19:40:41] Lou Netizen: I'm sure they can, Jenna: a lot of that stuff is automated. There's a whole industry around it.
A[19:40:43] Simon Linden: sure Jenna, they can put in whatever filters and blocks on network traffic that they want
A[19:40:50] Janet Rossini: agree it sounds like aws but the 499 is odd
A[19:41:19] Jenna Felton: when that is blocking, than you can not do anything but send the admins a note to unblock the SL address
A[19:41:32] Simon Linden: I don't know, Joe. In any case, there's always so much caching going on things might be stale
A[19:41:42] Lou Netizen: anyway: right now it really doesn't look like an SL problem just the 499 being a weird symptom of something upstream
A[19:42:16] Lou Netizen: I have my bot looking around still collecting data, we'll see if it finds anything weird
A[19:42:25] animats Resident: OK.
A[19:43:56] animats Resident: Sailors report more trouble.
A[19:44:03] animats Resident: Much anger on forums.
A[19:44:55] Janet Rossini: hm we are experimenting with region crossing non physical objects but not running them all the time
A[19:45:03] Qie Niangao: well, you know sailors, so high-strung, like golfers.
A[19:45:13] Simon Linden: hmmm it would be interesting to see if those reports match up with a recent updaet
A[19:45:27] Janet Rossini: they should lighten up, in the old days sailing in SL meant certain death.
A[19:45:39] animats Resident: I'm not seeing worse behavior with motorcycles.
A[19:46:09] Kelly Shergood: I've not heard any significant complaint increases with aircraft
A[19:46:22] Simon Linden: did you notice motorcycles changing when the original update a few weeks back went out?
A[19:46:41] animats Resident: No, motorcycles seemed to behave about the same.
A[19:47:11] animats Resident: About one half-unsit per half hour of riding.
A[19:47:31] animats Resident: How.s the real fix for half-unsits coming?
A[19:47:33] Inara Pey: Speaking as those living on Blake Sea and regularly out on motor boats, sailing craft and (now) hovercraft) ... we've both felt things are a lot more predictable at sea with crossings than previously.
A[19:47:56] Inara Pey: We've even dared to try wakeboarding again.... (!)
A[19:48:19] Simon Linden: lol I'll bet SL can give you a wild ride doing that sometimes
A[19:49:15] animats Resident: We really need the half-unsit fix. That's the one that totally destroys immersion.
A[19:49:29] Simon Linden: I am getting back to that work this week, hopefully. The viewer and server really should be able to understand a crossing didn't work properly
A[19:49:41] Inara Pey: You're welcome to join us, Simon - it's fun .... unless things really do go sideways. literally as well as figuratively...
A[19:49:55] animats Resident: Yes. Do you log region crossing fails server side?
A[19:50:26] Janet Rossini: in our crossing stuff we've seen objects disappear entirely. is that a known behavior if a handoff fails?
A[19:50:48] animats Resident: I've seen that too. Object disappears and is not returned.
A[19:51:02] Rex Cronon: r they temp objects?
A[19:51:06] Janet Rossini: no
A[19:51:10] animats Resident: No.
A[19:51:17] Qie Niangao: or scripted die-at-edge?
A[19:51:18] Lou Netizen: if that ever gets fixed will I have heaps of cars from 2010 land in my inventory? Because I lost a *lot* of them.
A[19:51:19] Simon Linden: yes Joe but I'm adding more detailed checks and reporting about all your attachments and sending updates
A[19:51:28] Janet Rossini: but they'll be at like -.99 and step further in to trigger the handoff
A[19:51:57] Simon Linden: sorry Lou, there won't be any spontaneous return of classic cars
A[19:52:03] Jenna Felton: can an object crossing region disappear when the leaving region errornously thinks it went off world?
A[19:52:06] animats Resident: Ah, good, Simon. First step is to detect and log. Then you have something to debug with.
A[19:52:06] Lou Netizen: praise be ;)
A[19:52:35] animats Resident: I have seen vehicles disappear in a half-unsit by going underground.
A[19:53:08] Rex Cronon: maybe they r left in the old sim?
A[19:53:23] animats Resident: Their scripts stop reporting to our server, but they continue to play their motion sounds and run their wheel animations.
A[19:53:47] Janet Rossini: yes i've seen that joe they even ownersay me
A[19:54:00] Simon Linden: and there's often different ways things can disappear ... sometimes it's a matter of you not being able to see it in the region
A[19:54:01] animats Resident: I've found them underground and pulled them up with edit.
A[19:54:03] Janet Rossini: but those ones usually come back
A[19:54:18] Simon Linden: or it really isn't there - it got returned, or somehow the data was lost on transfer
A[19:54:41] Rex Cronon: it was put back in user inventory?
A[19:54:44] Janet Rossini: we've had objects walking back and forth across the border (which by the way seems to generate lag) ...
A[19:54:51] Janet Rossini: and then after hours or days, they're just gone
A[19:54:52] animats Resident: Right. But always, the vehicle and passenger become separated.
A[19:54:52] Simon Linden: maybe Rex
A[19:55:05] Simon Linden: yes Janet, region crossings cause lag events
A[19:55:16] Janet Rossini: yes :)
A[19:55:27] Janet Rossini: moving a script etc must be tricky and costly
A[19:56:08] animats Resident: Yes. I have an instantaneous lag meter, which reports brief lags of over 200ms. Every time a YavaPod crosses the sim boundary, it trips.
A[19:56:09] Rex Cronon: if there was a lsl functions: llTurnOffScriptsOnSimCrossing :)
A[19:56:40] Janet Rossini: well i'm moving under script setpos equivalent so i can't live with that :)
A[19:56:48] Simon Linden: if you dig into the viewer open-source code, you'll see that on a region crossing there isn't a good, final exchange of messages between the viewer and 2 regions of "all done, you are now on region B with all your attachments and vehicle"
A[19:57:11] Janet Rossini: more like bye, good luck, may ebbe be with you?
A[19:57:19] Simon Linden: if they can't agree on that, something failed ... and it should log what's going on
A[19:57:57] animats Resident: I've been meaning to do that. Is there a document that really describes the state machines on sim and viewer side? I've seen the message sequence for the good case.
A[19:58:03] Rider Linden: I'm going to sneak out. Thank you everyone.
A[19:58:10] Yuzuru Jewell: Thank you, Rider.
A[19:58:11] Janet Rossini: bye rider
A[19:58:14] Simon Linden: in short Joe, no
A[19:58:16] Kennylex Luckless: take care Simon
A[19:58:17] Rex Cronon: maybe we shouldn't be able to just cross sims, maybe we should have something like a portal/stargate?
A[20:01:37] Simon Linden: I would be interested in experimenting with region crossings where we give up trying to make them perfectly invisible, with motion prediction and all that
A[20:01:43] Jenna Felton: but i think to solve region crossing problems you would need to make a region to control the vehicle on the other region while it is being transferred to that region. but that needs some revamping the SL :)