How long

From Lunar Linux
(Difference between revisions)
Jump to: navigation, search
(→Estimating compile time)
m (→Estimating compile time)
 
(10 intermediate revisions by 5 users not shown)
Line 4: Line 4:
 
* To recompile all my modules?
 
* To recompile all my modules?
  
−
These questions get asked a lot by new lunar users and it is a very logical question: Lunar is after all a source distro and requires you to compile all the applications from source, which can sometimes take extremely long!
+
These questions get asked a lot by new lunar users and it is a very logical question: Lunar is after all a source distro and requires you to compile all the applications from source, which can sometimes take an extremely long time!
  
  
Line 20: Line 20:
 
* ''System architecture and components''
 
* ''System architecture and components''
 
Needless to say that a compile on a SATA hard disk is a lot faster than on a ATA-33 system. Especially with larger modules this is a very important factor.
 
Needless to say that a compile on a SATA hard disk is a lot faster than on a ATA-33 system. Especially with larger modules this is a very important factor.
 +
 +
* ''Parallel make jobs''
 +
If you enable multiple parallel make jobs using '''lunar / Options / Optimize Architecture''' and you have a machine with more than one processor or core, most modules will build in a lot less time. Similarly if you have multiple machines and make use of '''distcc''' to share the build processes among your systems.
  
  
 
===Some basic numbers===
 
===Some basic numbers===
  
−
The following numbers are from a Pentium 4 laptop system with 256 MB, running at 1.8GHz. This is a fairly normal system and if you wish to compare it with your own system, you should keep in mind that this system might be relatively slower or faster based on the influencing factors mentioned earlier.
+
The following numbers are from a Pentium 4 system running at 2.9 GHz, with 1 GB of memory, which /proc/cpuinfo rates at 5864 bogomips. This was a fairly normal system back in 2005 and if you wish to compare it with your own system, you should keep in mind that this system might be relatively slower or faster based on the influencing factors mentioned earlier.
  
 
The modules listed below are very common modules, so they should give you some good insight in how long it takes to compile and install something. Based on the size you can make rough estimates for new modules that you wish to install.
 
The modules listed below are very common modules, so they should give you some good insight in how long it takes to compile and install something. Based on the size you can make rough estimates for new modules that you wish to install.
  
 
  module      source size    installed size    compile time
 
  module      source size    installed size    compile time
−
  gcc         27.6         257.3              120:45
+
  gcc           61540 KB        97188 KB         80m35s
−
XOrg        48.2          93.8              68:07
+
  glibc         15340 KB       138268 KB          32m47s
−
  glibc      13.5          64.7              47:09
+
  gtk+-2       18672 KB        64884 KB          15m56s
−
  gtk+-2     11.8          34.5              26:08
+
  perl         14792 KB        56844 KB           9m43s
−
  perl         9.8           52.3              17:50
+
  
−
As you clearly can see there is some correlation between installed size and compile time, but there are many outliers in this trend: gtk+-2 comes with a lot of bitmaps and documentation, while the XOrg tarball is counted without documentation. The gcc compiler itself has a very elaborate bootstrap phase that takes almost half the time it takes to compile the target compiler. Many many module-specific factors are at work here.
+
However, it is not easy to see whether there is any correlation between source size, installed size and compile time from such a small sample. Not all modules are built and installed in the same way. The gcc compiler has a very elaborate bootstrap phase that takes almost half the time it takes to compile the target compiler. The glibc shown here includes all possible locales. gtk+-2 comes with a lot of bitmaps and documentation. Many many module-specific factors are at work here.
  
  
 
===Estimating compile time===
 
===Estimating compile time===
  
−
If you correlate the size of the installed product or the source code size (in compressed form easily available), you get a very loose correlation shown in the graph below:
+
If we plot the compile time against the size of the installed product or the source code size (in compressed form easily available), we get a very loose correlation shown in the graph below, where the straight lines are produced using gnuplot's ''fit'' function:
  
−
http://foo-projects.org/~sofar/byinstallsize.png
+
http://lunar-linux.org/~engelsman/how-long/module-compile-times-raw.jpg
  
−
Above is correlated by install size (uncompressed size in kB against compile time in seconds), below is correlated against the size of the compressed source code tarballs (same scale).
+
The plot shows that there are an awful lot more small modules than large ones in the 400+ modules installed on this machine, but the range of values obscures any details. If we plot the same basic data using LOG values, we get the following graph:
  
−
http://foo-projects.org/~sofar/bysourcesize.png
+
http://lunar-linux.org/~engelsman/how-long/module-compile-times-log.jpg
  
−
As you can see there is a faint correlation possible. Most applications install about 10kB of product in 1 second. The other way around you can reason that it takes 1 second to compile 10kB of (compressed) source code. This allows you to pre-estimate the compile time, assuming that your system is the same speed as my reference system.
+
As you can see there is a faint correlation possible using either installed size or source size, but it's more interesting to predict compile times based on source size because once we have the installed size we already know the compile time anyway. Where the source size line crosses the x-axis, LOG(Compile time) is 0.0 and LOG(Source size) is almost 3.0, so Compile time is 1s and Source size is approx 20KB! [Note: that's LOG to base e, not LOG to base 10] This gives you a very crude compile speed of 20 KB/s, and allows you to pre-estimate the compile time, assuming that your system is the same speed as my reference system.
  
−
We can do some more elaborate math on this and calculate the "index" for all modules. The index of a module would be easily calculated by the amount of time it took for lunar to compile and install 10kB of compressed source code. So we divide the total compile/install time (seconds) by the number of kilobytes that the compressed source package was:
+
In fact, if I divide the total size of all of the sources by the total compile time, the average compile speed is 24.8 KB/s, so the estimate isn't too far off.
  
−
gcc  took 8025 to compile + install 26952 compressed kB of source code == 3.35
+
===Indexing modules===
−
XOrg took 4087 to compile + install 47656 compressed kB of source code == 11.6
+
  
−
If we do this for all modules we end up with the average index of about 27, which means that on average, lunar compiles and installs 27kB of compressed source code per second. Since the size of my /var/spool/lunar is about 1.4gB, it would take me about 51851 seconds to compile and install all of this source code files. That's about 14.5 hours! Of course that number really is not right - we haven't taken into account that larger modules might have a lower index and thus take much longer to compile.
+
We can do some more elaborate math on this and calculate an "index" for all modules. The index of a module could be easily calculated by dividing the actual compile speed for a module by the average compile speed:
  
 +
gcc    - 61520 KB of compressed source code compiled and installed in 80m35s, i.e. speed is 12.7 KB/s and index is 0.51
 +
glibc  - 15340 KB of compressed source code compiled and installed in 32m47s, i.e. speed is  7.8 KB/s and index is 0.31
 +
gtk+-2 - 18672 KB of compressed source code compiled and installed in 15m56s, i.e. speed is 19.5 KB/s and index is 0.80
 +
perl  - 14792 KB of compressed source code compiled and installed in  9m43s, i.e. speed is 25.4 KB/s and index is 1.02
  
−
===Indexing modules===
+
We could, of course, do this for all modules, but this would involve someone actually downloading and installing every one of them, and this just isn't realistic as there are currently more than 3600 modules in the moonbase. [I tried to download all modules once using '''lget -a''' and it came to 15GB. Using the approximate compile speed of 20KB/s it would take more than 8 days to build everything]
 +
 
 +
 
 +
===From ISO install to desktop===
 +
 
 +
Just out of curiosity, and for information to others, I decided to time how long it took to install a "minimal" desktop system from scratch from the 1.6.5 ISO on a relatively new system and a 9-year-old relic. I might add this 5-year-old box in the future.
 +
 
 +
It's clear that you probably don't want to install Lunar on really old hardware, unless you have a lot of patience, or can use <code>distcc</code>.
 +
 
 +
 
 +
{| border="1"
 +
! System
 +
! CPUs*bogomips [1]
 +
! RAM
 +
! Disk [2]
 +
! Install
 +
! lin gcc...
 +
! lin linux-2.6
 +
! lunar update
 +
! lin XOrg7
 +
! lin xfce4
 +
! TOTAL
 +
|- align="center"
 +
| Acer AX5810
 +
| 4 * 4988
 +
| 6 Gb
 +
| 1681 / 129 MB/s
 +
| 0h11
 +
| 1h01
 +
| 0h12
 +
| 0h20
 +
| 0h23
 +
| 0h14
 +
! 2h21
 +
|- align="center"
 +
| Vaio FX401
 +
| 1 * 1600
 +
| 512Mb
 +
| 179 / 17 MB/s
 +
| 0h18
 +
| 11h26
 +
| 3h37
 +
| 2h44
 +
| 3h12
 +
| 2h15
 +
! 23h32
 +
|}
 +
 
 +
# <code>grep bogomips /proc/cpuinfo</code>
 +
# <code>hdparm -t -T /dev/sdX</code>
 +
 
 +
 
 +
The basic procedure:
 +
* install from the ISO, and take the pre-compiled kernel, and reboot
 +
* run <code>lin moonbase</code>
 +
* run <code>lunar</code> and set number of parallel makes to 4 in ''Options / Optimize Architecture''
 +
* do the <code>lin gcc glibc gcc bash coreutils tar wget</code> dance from <code>man lfirsttime</code>
 +
* use <code>lin linux-2.6</code> to explicitly build a new kernel, and reboot
 +
* run <code>lunar update</code> to rebuild any outdated parts of the core system
 +
* run <code>lin XOrg7</code> to build X
 +
* run <code>lin xfce4</code> to build Xfce4
 +
 
 +
After the first reboot I primed the system by copying sources from /var/spool/lunar on another up-to-date machine to avoid download delays and glitches.
 +
 
 +
When asked questions about optional depends, etc. I simply hit return to accept the default values. I did not reconfigure the kernel. I used <code>lin --deps</code> for the XOrg7 and xfce4 builds to pre-answer the questions before kicking off the real builds so I could leave them unattended without adding delays for prompts to timeout.
  
−
We should be able to index the numbers of all modules right now, so it pays off to pay a closer look at the numbers a bit. Most importantly, the 27 index is averaged across all modules, but we know that this number is off for bigger modules. For modules larger than 100kB source size, it's already 32, and for modules over 1000kB, it's up to 52. This seriously changes perspective and shows you that smaller modules take relatively longer to install, and this is not surprising: the overhead is usually much larger for them, as all the autoconf and administrativa needs to be performed for smaller modules as well.
+
NOTE: this procedure is for timing purposes only, I still have to go back and install the correct drivers for X, etc.

Latest revision as of 22:57, 24 September 2010

Contents

How long does it take...

  • To install a specific module?
  • To install the ISO?
  • To recompile all my modules?

These questions get asked a lot by new lunar users and it is a very logical question: Lunar is after all a source distro and requires you to compile all the applications from source, which can sometimes take an extremely long time!


Influencing Factors

  • System(CPU) speed

Double the CPU MHz, and you double the amount of instructions that a system can make per time unit. So, as a rule, a faster CPU results in faster compiles.

  • Memory size

Compiling may require large amounts of memory, especially when compiling C++ applications. Generally, a compilation will go 50% faster if you have double the memory. So, if you only have 128MB of memory, you will speed up your compiles by 225% by increasing the amount of memory to 512MB. This might be worth the trip to the hardware store.

  • Idle time

Compiling takes a long time. Every second that your system is doing something else, especially interactive tasks, reduces the amount of time that your system can compile applications.

  • System architecture and components

Needless to say that a compile on a SATA hard disk is a lot faster than on a ATA-33 system. Especially with larger modules this is a very important factor.

  • Parallel make jobs

If you enable multiple parallel make jobs using lunar / Options / Optimize Architecture and you have a machine with more than one processor or core, most modules will build in a lot less time. Similarly if you have multiple machines and make use of distcc to share the build processes among your systems.


Some basic numbers

The following numbers are from a Pentium 4 system running at 2.9 GHz, with 1 GB of memory, which /proc/cpuinfo rates at 5864 bogomips. This was a fairly normal system back in 2005 and if you wish to compare it with your own system, you should keep in mind that this system might be relatively slower or faster based on the influencing factors mentioned earlier.

The modules listed below are very common modules, so they should give you some good insight in how long it takes to compile and install something. Based on the size you can make rough estimates for new modules that you wish to install. 

module      source size    installed size    compile time
gcc           61540 KB        97188 KB          80m35s
glibc         15340 KB       138268 KB          32m47s
gtk+-2        18672 KB        64884 KB          15m56s
perl          14792 KB        56844 KB           9m43s

However, it is not easy to see whether there is any correlation between source size, installed size and compile time from such a small sample. Not all modules are built and installed in the same way. The gcc compiler has a very elaborate bootstrap phase that takes almost half the time it takes to compile the target compiler. The glibc shown here includes all possible locales. gtk+-2 comes with a lot of bitmaps and documentation. Many many module-specific factors are at work here.


Estimating compile time

If we plot the compile time against the size of the installed product or the source code size (in compressed form easily available), we get a very loose correlation shown in the graph below, where the straight lines are produced using gnuplot's fit function:

http://lunar-linux.org/~engelsman/how-long/module-compile-times-raw.jpg

The plot shows that there are an awful lot more small modules than large ones in the 400+ modules installed on this machine, but the range of values obscures any details. If we plot the same basic data using LOG values, we get the following graph:

http://lunar-linux.org/~engelsman/how-long/module-compile-times-log.jpg

As you can see there is a faint correlation possible using either installed size or source size, but it's more interesting to predict compile times based on source size because once we have the installed size we already know the compile time anyway. Where the source size line crosses the x-axis, LOG(Compile time) is 0.0 and LOG(Source size) is almost 3.0, so Compile time is 1s and Source size is approx 20KB! [Note: that's LOG to base e, not LOG to base 10] This gives you a very crude compile speed of 20 KB/s, and allows you to pre-estimate the compile time, assuming that your system is the same speed as my reference system.

In fact, if I divide the total size of all of the sources by the total compile time, the average compile speed is 24.8 KB/s, so the estimate isn't too far off.

Indexing modules

We can do some more elaborate math on this and calculate an "index" for all modules. The index of a module could be easily calculated by dividing the actual compile speed for a module by the average compile speed: 

gcc    - 61520 KB of compressed source code compiled and installed in 80m35s, i.e. speed is 12.7 KB/s and index is 0.51
glibc  - 15340 KB of compressed source code compiled and installed in 32m47s, i.e. speed is  7.8 KB/s and index is 0.31
gtk+-2 - 18672 KB of compressed source code compiled and installed in 15m56s, i.e. speed is 19.5 KB/s and index is 0.80
perl   - 14792 KB of compressed source code compiled and installed in  9m43s, i.e. speed is 25.4 KB/s and index is 1.02

We could, of course, do this for all modules, but this would involve someone actually downloading and installing every one of them, and this just isn't realistic as there are currently more than 3600 modules in the moonbase. [I tried to download all modules once using lget -a and it came to 15GB. Using the approximate compile speed of 20KB/s it would take more than 8 days to build everything]


From ISO install to desktop

Just out of curiosity, and for information to others, I decided to time how long it took to install a "minimal" desktop system from scratch from the 1.6.5 ISO on a relatively new system and a 9-year-old relic. I might add this 5-year-old box in the future.

It's clear that you probably don't want to install Lunar on really old hardware, unless you have a lot of patience, or can use distcc.


System CPUs*bogomips [1] RAM Disk [2] Install lin gcc... lin linux-2.6 lunar update lin XOrg7 lin xfce4 TOTAL
Acer AX5810 4 * 4988 6 Gb 1681 / 129 MB/s 0h11 1h01 0h12 0h20 0h23 0h14 2h21
Vaio FX401 1 * 1600 512Mb 179 / 17 MB/s 0h18 11h26 3h37 2h44 3h12 2h15 23h32
  1. grep bogomips /proc/cpuinfo
  2. hdparm -t -T /dev/sdX


The basic procedure:

  • install from the ISO, and take the pre-compiled kernel, and reboot
  • run lin moonbase
  • run lunar and set number of parallel makes to 4 in Options / Optimize Architecture
  • do the lin gcc glibc gcc bash coreutils tar wget dance from man lfirsttime
  • use lin linux-2.6 to explicitly build a new kernel, and reboot
  • run lunar update to rebuild any outdated parts of the core system
  • run lin XOrg7 to build X
  • run lin xfce4 to build Xfce4

After the first reboot I primed the system by copying sources from /var/spool/lunar on another up-to-date machine to avoid download delays and glitches.

When asked questions about optional depends, etc. I simply hit return to accept the default values. I did not reconfigure the kernel. I used lin --deps for the XOrg7 and xfce4 builds to pre-answer the questions before kicking off the real builds so I could leave them unattended without adding delays for prompts to timeout.

NOTE: this procedure is for timing purposes only, I still have to go back and install the correct drivers for X, etc.

Retrieved from "http://doc.lunar-linux.org/index.php?title=How_long&oldid=3278"
Personal tools
Namespaces
Variants
Actions
Wiki Navigation
Project Sites
Toolbox