**** BEGIN LOGGING AT Sun Feb 18 12:41:29 2001

12:41:29 -->	RobertFitzsimons (bob@194.125.56.12) has joined #jos
12:41:29 ---	Topic for #jos is JOS The Java Operating System
12:41:29 ---	Topic for #jos set by jewel at Sun Feb 18 12:35:07 2001
12:41:46 <RobertFitzsimons>	Hi Jewel.
12:47:53 <jewel>	Hey RobertFitzsimons
12:48:10 <jewel>	Once again I'm glad to see you here, but this time I have to go :)
12:48:18 <RobertFitzsimons>	:(
12:49:58 *	jewel thinks about cancelling his other committment
12:50:57 <jewel>	ok, hang for 5 minutes, if I don't come back then I won't be here for a few hours
12:51:03 <RobertFitzsimons>	Ok.
12:51:06 <jewel>	but I might be back in 5 minutes 
12:54:13 <--	jewel has quit (Ping timeout for jewel[pta-dial-196-31-185-110.mweb.co.za])
12:56:07 -->	jewel (jewel@pta-dial-196-31-186-36.mweb.co.za) has joined #jos
12:57:04 <--	jewel has quit (leaving)
12:59:06 -->	jewel (jewel@pta-dial-196-31-186-36.mweb.co.za) has joined #jos
12:59:22 <jewel>	ah
12:59:26 <jewel>	much rearranging later
12:59:36 <RobertFitzsimons>	You got out of it then?
12:59:49 <jewel>	yeah I cancelled
13:00:09 <jewel>	it's amazing how one decision can change the future course of the day so much
13:00:35 <RobertFitzsimons>	What were your plans before you changed them?
13:01:11 <jewel>	I was going to Johannesburg (about 40 minutes drive) to visit some friends, maybe go the planetarium and then go eat waffles
13:01:40 <jewel>	but the plans were only tens minutes old before I changed them :-)
13:01:47 <RobertFitzsimons>	;)
13:03:23 <jewel>	So what have you been up to?
13:04:20 <RobertFitzsimons>	All last week I was very busy, I was working most night until 11.  So I didn't get a chance to work on JOS.
13:04:52 <RobertFitzsimons>	My plan for today is to work on memory management and dynamic loading.
13:05:37 <jewel>	Cool
13:05:52 <jewel>	I've probably asked you before, but what kind of software do you develop for your work?
13:07:27 <RobertFitzsimons>	We have a middle-ware product which provides message parsing and connectivity to mainframe systems, this product is mainly used by the travel industry.
13:08:32 <jewel>	various mainframe systems or just one type?
13:10:23 <RobertFitzsimons>	Lot's, we don't deal directly with the mainframes, they usally have IP level protocals then we deal with.
13:11:01 <jewel>	Yuck
13:11:15 <jewel>	So it's all about transferring info between a whole bunch of incompatible systems
13:12:27 <RobertFitzsimons>	Yes, that's part of it, but mostly we only deal with one system at a time.
13:13:39 <jewel>	Ok
13:20:10 <RobertFitzsimons>	Were you able to do any work on teaseme over the last week?
13:21:10 <jewel>	Yep I've done some work
13:21:21 <jewel>	Most of it is just related to the process support stuff though
13:21:31 <jewel>	No new code or anything
13:25:30 <jewel>	Ah man!
13:25:45 <jewel>	I'm trying to get some debug output from the Java code
13:25:54 <jewel>	but there is a bug which chops off everything but the first letter
13:26:47 <RobertFitzsimons>	Will teaseme have a problem with RJK only being able to allocate memory in multiples of the system pagesize?
13:27:07 <jewel>	Yes
13:27:21 <jewel>	I mean, if it's more memory than requested, then it's fine
13:27:23 <jewel>	but not less
13:28:49 <RobertFitzsimons>	It wouldn't be to hard to add a wrapper which will allocate memory in smaller blocks.
13:29:19 <jewel>	well leave it like that for now, just make sure it rounds up the requests
13:29:40 <RobertFitzsimons>	Yes, thats what it'll do.
13:31:02 <RobertFitzsimons>	Also do you know how big the stack is normaly?  Currently I only allocate 16KB, but I could change it to be user defined.
13:33:21 <jewel>	I'm not sure
13:33:40 <jewel>	I think glibc does some kind of mmap, so that it doesn't actually allocate more stack until necessary
13:33:48 <jewel>	but I think it's big, like 2MB
13:37:10 <RobertFitzsimons>	Yeah, that's what I was thinking.  The problem with RJK is that all virutal memory shares the same 4GB VM space so the number of virutal allocations is very limited.
13:38:06 <RobertFitzsimons>	I.e. you can only have a maximum of 2000 threads with a 2 MB stack size.
13:38:22 <jewel>	It's difficult to say how deep the stack will grow
13:38:44 <jewel>	It's fairly safe to say that we won't need more than 4000 threads in the near future
13:40:00 <RobertFitzsimons>	Maybe, though JOS should be a very good platform for running Java servers, which could have lots and lots of threads.
13:40:19 <jewel>	Yes
13:40:38 <jewel>	In that case we'll first have to figure out how deep the average thread grows
13:40:53 <jewel>	Remember these are the C thread stacks
13:41:08 <jewel>	teaseme creates Java stacks in memory with a linked-list approach
13:41:25 <RobertFitzsimons>	That'll help.
13:49:19 *	jewel finally homes in on the bug
**** BEGIN LOGGING AT Sun Feb 18 14:05:01 2001

14:05:01 -->	RobertFitzsimons (bob@194.125.56.12) has joined #jos
14:05:01 ---	Topic for #jos is JOS The Java Operating System
14:05:01 ---	Topic for #jos set by jewel at Sun Feb 18 12:35:07 2001
14:05:28 *	RobertFitzsimons wonders why his computer just turned off.
14:07:17 <jewel>	lost power?
14:08:27 <RobertFitzsimons>	I could have, I don't have the battery in my laptop at the moment.  But the server rack beside me didn
14:08:34 <RobertFitzsimons>	't go down.
14:09:44 <jewel>	loose connection?
14:10:08 <RobertFitzsimons>	Yeah maybe.
14:10:40 *	jewel notes that if you want a java method to return early without commenting out the rest of the code, just go: if(true) return;
14:11:05 *	jewel also notes that jikes +U will check dependencies for your java files while compiling
14:11:31 <RobertFitzsimons>	Yeah, jikes is pretty good.
14:14:00 *	jewel succeeds in running more JOS processes
14:16:31 <jewel>	I'm so glad I created the userspace port
14:16:36 <jewel>	debugging is so much faster and easier
14:17:43 <jewel>	is sleep() thread-safe ?
14:17:52 <RobertFitzsimons>	Yes, it's much easier.  Or you could use user mode linux?
14:18:18 <RobertFitzsimons>	It should be, where are you calling it from, the kernel?
14:19:38 <jewel>	I mean in user-space
14:20:03 <jewel>	When I call it, it seems to interfere with pthread_create 
14:20:09 <jewel>	but only in the debugger
14:22:12 <RobertFitzsimons>	The problem is in the debugger (gdb), it doesn't handle threads very well.
14:22:28 <jewel>	I suspected so
14:25:52 <jewel>	On another topic:
14:26:09 <jewel>	If I want to compile something on debian which depends on the CVS version of libart
14:26:27 <jewel>	How do I make sure the program sees the local copy of libart and not the debian pkg
14:30:40 <RobertFitzsimons>	I think you'll need to change/set the INCLUDES/LD_LIBRARIES_PATH environment variables.
14:32:10 <jewel>	ah, ok, I will look at that
14:48:53 <jewel>	how is it going?
14:50:15 <RobertFitzsimons>	I totaly wiped out the old VM code and I'm changing some of the api functions for mapping memory.
14:51:39 <jewel>	VM == virtual memory?
15:00:11 <RobertFitzsimons>	Sorry, yes.
15:19:44 <jewel>	Mmm, when should the server processes exit?
15:20:39 <jewel>	I assume it will be easy to change teaseme to use your memory-manager
15:20:53 <jewel>	The most interesting part is the Java heaps
15:21:06 <jewel>	They use valloc and will be large blocks (> 0.5MB each)
15:21:37 <jewel>	Due to the nature of a Java heap they will fill up unless the process exits fairly early
15:22:27 <RobertFitzsimons>	You might want to look at the JNI spec for that, it says something about only exiting when there are only daemon threads left.
15:22:43 <RobertFitzsimons>	Yeah, it shouldn't be to hard.
15:23:02 <jewel>	hmm the non-daemon thread doesn't really apply here that way
15:23:27 <jewel>	You see, the processes don't need to have any threads running to stay alive
15:23:28 <RobertFitzsimons>	Then it's after the last thread has exited.
15:26:59 <jewel>	But the way it is now, they don't need to have any threads running to service a request
15:28:51 <jewel>	So if they have to keep a thread running, what should it do?
15:28:58 <RobertFitzsimons>	Oh, process server.
15:31:01 <RobertFitzsimons>	Can you explain how processes work?
15:31:02 <jewel>	server process
15:31:26 <jewel>	well you create a new process by calling an 'IPC' method on the ProcessServer
15:31:44 <jewel>	you find the ProcessServer by looking at a special field in teaseme.system.Servers
15:32:29 <jewel>	The ProcessServer then creates the creates the process structures, creates a new thread and threadgroup and starts it executing the main() method of a class you specified in the IPC call
15:32:58 <jewel>	The ProcessServer returns a object to you of type JOSProcess
15:33:23 <jewel>	The process currently runs until the first thread exits (eventually till all non-daemon threads exit)
15:33:45 <jewel>	A server process is very similar
15:33:55 <jewel>	But how do the server processes wait for a system shutdown?
15:34:06 <jewel>	How do we do a shutdown with a console ! ?
15:34:44 <jewel>	 s /with/without
15:37:53 <RobertFitzsimons>	Maybe you need to throw a special exception (SystemShutdownException?) on every thread to kill them, then destory each process, and reboot?
15:38:45 <jewel>	We can do that if server processes refuse to exit
15:38:58 <jewel>	But how can they be notified (politely) that shutdown time has arrived
15:39:14 <jewel>	The problem is that in teaseme the system starts up and shuts down immediately
15:39:23 <jewel>	should I wait ten seconds before shutting down?
15:40:58 <RobertFitzsimons>	I'm a little confussed between the JVM process and these other processes.  Wait ten seconds for which one to shut down?
15:42:07 <jewel>	I mean the whole teaseme (as an OS), waits 10 seconds before it tells everything to shut down
15:43:55 <RobertFitzsimons>	Ok, provide a hook which will shutdown the system when called.  For the moment just have a timer which will call this hook after 10 seconds, we can then do it correctly when we have a running system.
15:45:10 <jewel>	then the other complication is I don't think I have an accurate Thread.sleep() in kernel mode
15:49:19 <RobertFitzsimons>	Is it in cvs?
15:52:51 <jewel>	my latest changes aren't in CVS yet
15:53:06 <jewel>	have you though of putting RJK in CVS?
15:55:28 <RobertFitzsimons>	A little, I want to wait until it's ready for a release, which won't be for a few week.
15:55:40 <RobertFitzsimons>	BRB, going for food.
15:57:22 <jewel>	ok
16:09:31 <RobertFitzsimons>	I'm back.
16:10:19 <jewel>	wb
16:14:27 <jewel>	I'm reading this right now: http://www.toad.com/gnu/whatswrong.html
16:26:52 <jewel>	may I look at your api?
16:27:40 <RobertFitzsimons>	Currently it's the same as in the kinterface.h file.
16:29:33 <jewel>	what is the diff between the linear and virtual functions?
16:33:43 <RobertFitzsimons>	Linear is just another name for physical, physical memory is used by for dma and by other hardware.
16:33:59 <jewel>	ah ok
16:34:09 <jewel>	so it has nothing do with linear as in "straight"
16:34:23 <RobertFitzsimons>	Yes.
16:34:38 <jewel>	will we be able to prevent certain memory from being paged out?
16:35:26 <jewel>	I read some more of that RTJ spec
16:35:31 <jewel>	It looks very good
16:35:50 <jewel>	I haven't read it thoroughly yet, but I think we will need to implement a lot of the things they have there
16:36:11 <RobertFitzsimons>	My plan was to only allow virtual memory to be paged out.
16:36:47 <jewel>	(that's what I mean, but can we lock certain vmem pages)
16:37:09 <jewel>	I think the technical term is 'pin'
16:38:19 <jewel>	How big are we going to make pages?
16:39:03 <RobertFitzsimons>	We can add a new function, where do you think it'll be used?
16:39:17 <jewel>	To pin the pages?
16:39:22 <RobertFitzsimons>	Yes.
16:39:33 <jewel>	This would useful for code that has to be in memory
16:39:53 <jewel>	Whatever that may be
16:40:07 <jewel>	if the page handler is written in Java, that code should be locked in for example
16:40:29 <RobertFitzsimons>	The current page size for on a i386 is 4096 bytes, it defined in a constant KPAGESIZE.
16:40:49 <jewel>	Mmm, on 64-bit boxes are we likely to encouter 8K?
16:41:51 <RobertFitzsimons>	Yes, it depends on the processor.  There is also an option for a 4MB page size on pentiums and above.
16:42:16 <jewel>	Is that ever used?
16:42:30 <jewel>	It can't be intermixed with another size can it?
16:43:18 <RobertFitzsimons>	I think you can mix it, but I'd have to check.
16:44:04 <jewel>	I remember once noticing that Ppros could address a lot more memory than other cpus
16:45:07 <RobertFitzsimons>	Yes, I belive there is a special mode for addressing 64GB of memory.
16:45:49 <jewel>	so what happens there?
16:46:02 <jewel>	do they have 36 bit addresses or something?
16:48:18 <RobertFitzsimons>	Yes, there is a special register which contains the extra 4 bits, so a linux process is only able access 4GB at a time.  It'll probaly be the same of JOS.
16:48:58 <jewel>	I've read a few papers on object-centric virtual memory and persistence
16:49:38 <jewel>	The fact that page sizes are fixed make a serious obstacle for object-oriented systems
16:50:03 <jewel>	Also the fact that so many objects are so small (200 bytes)
16:50:14 <jewel>	you have to intelligently group them into a page
16:52:31 *	jewel has to add up the marks for a stack of 100 high-school maths papers
16:52:37 <RobertFitzsimons>	Yes, the number of pages that are used by the internal data structures is very important to speed of the JVM.
16:53:11 <jewel>	never thought of that
16:53:30 <jewel>	I've done quick tests to see how big the average object is, but didn't look at the tClass structures
16:54:36 <jewel>	I read the technical report for SableVM (I think I pointed you to it a while ago)
16:54:55 <jewel>	In that he does his own VMM within the VM for those class structures to optimise memory usage
16:55:09 <RobertFitzsimons>	Yes, I spent a lot of time thinking about the class data structure in ujvm.
16:56:34 <RobertFitzsimons>	Basically, I was trying to have one allocated block which contained the whole class data structure.
16:57:01 <jewel>	With no separate allocations?
16:57:37 *	jewel finishes the maths papers
16:57:51 <RobertFitzsimons>	Well done.
16:58:53 <jewel>	I think that's pretty much what Etienne is doing in SableVM too
16:59:14 <RobertFitzsimons>	You can use separate allocations to build the data structure, but then you find the total size, and then copy it to the new memory block.
16:59:32 <jewel>	yep
17:03:14 <jewel>	Argh, the Java sematics break down in an OS environment
17:03:27 <jewel>	I wanted to make an object which processes would wait() on 
17:03:48 <jewel>	then at shutdown time the init process (or whatever) calls notify on the object
17:04:06 <jewel>	the problem is that any process in the system can call notify and shutdown the whole machine!
17:04:39 <jewel>	I could also require each server process to have a callback routine
17:04:48 <jewel>	(They all extend the same base class)
17:05:41 <RobertFitzsimons>	Yes, that's a better idea.
17:06:10 <RobertFitzsimons>	The other way would require an extra thread for every process.
17:07:15 *	jewel investigates the callback method
17:18:06 <jewel>	I've decided to go to Johannesburg now
17:18:18 <jewel>	(after more invitations)
17:18:26 <jewel>	So I'll see you later
17:18:33 <RobertFitzsimons>	Sure, talk to you later.
17:18:42 <jewel>	When I finish my work on the shutdown stuff I'll upload to CVS and post on the list
17:19:04 <RobertFitzsimons>	Ok, great.
17:19:05 <jewel>	After that I'll also try compiling some parts of teaseme against RJK to get a feel for what we'll need to do for the port
17:19:34 <jewel>	Happy coding
17:19:38 <RobertFitzsimons>	Bye.
17:22:43 <--	jewel has quit (Ping timeout for jewel[pta-dial-196-31-186-36.mweb.co.za])
17:58:54 ---	You have left channel #jos
**** ENDING LOGGING AT Sun Feb 18 17:58:55 2001