rar better than zip

/usr/local/bin/unrar 

unrar x   *.rar  ./data  (uncompress *.rar  into folder  /data)

/usr/local/bin/zip -9 b1.zip ./data2/*    (use zip to compress /data/* into b1.zip)

install from ports  /usr/ports/archivers

http://jeremy.zawodny.com/blog/archives/000796.html

http://www-db.stanford.edu/~backrub/google.html

Where does this limit come from?

In a MyISAM table with dynamic (variable length) rows, the index file for the table (tablename.MYI) stores row locations using 32-bit pointers into the data file (tablename.MYD). That means it can address only 4GB of space.

This problem is both a historical artifact and an optimization. Back when MySQL was created, it wasn’t common to store that much data in a single table. Heck, for a long time 4GB was an entire hard disk and most operating system had trouble with files larger than 2GB. Obviously those days are gone. Modern operating systems have no trouble with large files and hard disks larger than 100GB are quite common.

From an optimization point of view, however, the 32-bit pointer still makes sense. Why? Because most people are running MySQL on 32-bit hardware (Intel/Linux). That will change as use of AMD’s Opteron becomes more widespread, but 32-bit will be the majority for the next few years. Using 32-bit pointers is the most efficient way to do this on 32-bit hardware. And even today, most MySQL installations don’t have tables anywhere near 4GB in size. Sure, there are a lot of larger deployments emerging. They’re all relatively new.

An Example

Here’s a table that you might use to store weather data:

mysql> describe weather;
+-----------+--------------+------+-----+------------+-------+
| Field     | Type         | Null | Key | Default    | Extra |
+-----------+--------------+------+-----+------------+-------+
| city      | varchar(100) |      | MUL |            |       |
| high_temp | tinyint(4)   |      |     | 0          |       |
| low_temp  | tinyint(4)   |      |     | 0          |       |
| the_date  | date         |      |     | 0000-00-00 |       |
+-----------+--------------+------+-----+------------+-------+
4 rows in set (0.01 sec)

To find its size limit, we’ll use SHOW TABLE STATUS

mysql> show table status like 'weather' \G
*************************** 1. row ***************************
           Name: weather
           Type: MyISAM
     Row_format: Dynamic
           Rows: 0
 Avg_row_length: 0
    Data_length: 0
Max_data_length: 4294967295
   Index_length: 1024
      Data_free: 0
 Auto_increment: NULL
    Create_time: 2003-03-03 00:43:43
    Update_time: 2003-03-03 00:43:43
     Check_time: 2003-06-14 15:11:21
 Create_options:
        Comment:
1 row in set (0.00 sec)

There it is. Notice that Max_data_length is 4GB. Let’s fix that.

mysql> alter table weather max_rows = 200000000000 avg_row_length = 50;
Query OK, 0 rows affected (0.03 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> show table status like 'weather' \G
*************************** 1. row ***************************
           Name: weather
           Type: MyISAM
     Row_format: Dynamic
           Rows: 0
 Avg_row_length: 0
    Data_length: 0
Max_data_length: 1099511627775
   Index_length: 1024
      Data_free: 0
 Auto_increment: NULL
    Create_time: 2003-06-17 13:12:49
    Update_time: 2003-06-17 13:12:49
     Check_time: NULL
 Create_options: max_rows=4294967295 avg_row_length=50
        Comment:
1 row in set (0.00 sec)

Excellent. Now MySQL will let us store a lot more data in that table.

http://ontosys.com/php/cache.html

PHP Cache Control

This note describes a scheme for allowing PHP pages to be cached by a browser.

Example files

cache_check.inc — Logic to support caching of dynamic pages

<?php
$if_modified_since = preg_replace('/;.*$/', '', $HTTP_IF_MODIFIED_SINCE);

$mtime = filemtime($SCRIPT_FILENAME);
$gmdate_mod = gmdate('D, d M Y H:i:s', $mtime) . ' GMT';

if ($if_modified_since == $gmdate_mod) {
    header("HTTP/1.0 304 Not Modified");
    exit;
}
header("Last-Modified: $gmdate_mod");
?>

The value of the HTTP If-Modified-Since header (if any) is available in $HTTP_IF_MODIFIED_SINCE. We check the date value in that header against the modification date of the executed PHP script file itself. If they are the same, we send a 304 response and quit.

Otherwise, we send a Last-Modified header with the file’s modification date.

cachable.php3 — An example cacheable dynamically-generated file

<?php // -*- sgml-parent-document: ("dummy.html" "html" "body" ()) -*-
include 'cache_check.inc';

if (isset($touch))  touch($SCRIPT_FILENAME);
$gmdate_now = gmdate('D, d M Y H:i:s') . ' GMT';
$now = time();

print "
<table>
<tr><td>if_modified_since</td><td>$if_modified_since</td></tr>
<tr><td>gmdate_mod</td><td>$gmdate_mod</td></tr>
<tr><td>gmdate_now</td><td>$gmdate_now</td></tr>
</table>
<p>
<a href=\"$SCRIPT_NAME\">Link to self</a>.<br>
<a href=\"$SCRIPT_NAME?touch=y&time=$now\">Update source file</a>.<br>
<a href=\"$SCRIPT_NAME?time=$now\">Link to self with varying URL</a>.
";
?>

This dynamic page simply generates some output for testing purposes. Note that the gmdate_now value will not appear to change if the browser uses the file from its cache or if the server sends back a 304.

RewriteCond %{REQUEST_FILENAME} /(.*)-f([0-9]*).html

<?php
$patterns = array (‘/(19|20)(\d{2})-(\d{1,2})-(\d{1,2})/’,
                   ‘/^\s*{(\w+)}\s*=/’);
$replace = array (‘\3/\4/\1\2′, ‘$\1 =’);
echo preg_replace($patterns, $replace, ‘{startDate} = 1999-5-27′);
?>

标题优化—怎样在Title中有效的使用关键词

文/流浪诗人

Placing keywords in titles of your web pages is essential in order to obtain good organic search engine rankings. Whatever text you place in the Title tags of a HTML page, it will appear in the title bar of a web browser such as FireFox and Internet Explorer. The title of a web page can be considered the most important place to put keywords in every search engine optimization plan.

把网站关键词放到网页的Title中是获得良好的有机排名的重点。无论你在HTML页面放入什么样的Title关键词，它都会出现在你的浏览器的顶部。在每个搜索引擎优化的计划中，网页Title被认为是最重要的使用网站关键词的地方。

The words that you place in your title determine how search engines may decide to rank your web page on their organic listings. Most search engines view the title of your web page as a direct indication of whether your website is relevant when someone does a search.

关键词在Title中的放置决定了搜索引擎怎样在它的有机列表中为你的网站排名。当搜索者输入关键词的时候，大多的搜索引擎会把你的网页Title看做是否和搜索内容相关的直接指示。

To optimize title effectively, you need to place unique title for each page of your website. The title of each page must be relevant to the content and keyword phrases used in that particular page itself.

有效的优化网页Title，你需要为你的每一个网页提供唯一的Title。每个网页的Title必须与网页的内容和使用的关键词相关。

One good technique of title optimization is to use a combination of words and phrases for different pages so as to drive different kinds of targeted traffic to your website. By doing this, you are able to optimize more keywords and thus getting more traffic for the different services and products offered on your website.

一个有效的Title优化原则是不同的网页使用不同的关键词组合，这样就能够为你的网站获得更多的不同的目标客户。这样做的好处是，你可以优化更多的关键词，从而给你的网站服务和产品带来更多的有效流量。

One common practice of website owners is that when they first design their websites, they place their business name in every title of every page. Not to say that placing business name in the title of a page is bad, but it should be done with combination of keywords that you intend to optimize. By just putting your business name, you are wasting valuable title space that can determine whether you rank number 1 or number 1000 in Google.

当站长在设计网站的时候，一个普遍的实践是，他们把自己网站的名称放在每一个网页的Title中。虽然这样做不能说不好，但是最好是在你的网页Title中加入你想要优化的关键词。如果只放网站的名称，就浪费了Title的价值，Title决定了网站在Google中的排名是第一位还是第1000位。

A better way to put your business name in your title is to place important keywords phrases at the front of the title and ending with your business name. Take note that the front of the title is very important for search engine optimization and make sure that you place the most important keywords at the front. In this case, you will stand a higher chance to rank high in search engines, as well as displaying your business name for branding purpose.

网页Title中加入网站名称的最好方法是把重要的关键词放在Title的前边，然后以网站名称结尾。注意，对于搜索引擎优化（SEO）来说Title的前半部分非常重要，所以一定要确保Title的前半部分使用的是最重要的关键词。这样，你就有机会在搜索引擎排名中获得良好的排名，同时对你的品牌战略也有很大的帮助。

With good planning of title and basic on-site optimization, you will be able to rank well for keywords if the competition for the selected keywords is not very tough.

如果关键词的竞争不是非常激烈，有一个好的Title优化和页面优化计划，会使你的每一个关键词都获得良好的排名。

Google checkout & paypal. What’s the difference ?

我们在很多情况下需要定制Apache服务器（HTTPD）的日志（LOG）。比如为了防止Apache LOG文件过大需要定制apache服务器的LOG文件名，
对于某些特定的URL比如图片等的访问不记录LOG等。

如何增加内存表的使用空间

this is how to change it with my.cnf
/usr/local/mysql/bin/mysqld_safe –user=mysql    -O max_heap_table_size=320M  &

默认的是16M，可以根据自己的需要，增加到需要的大小

How to optimize your sql performance?

Check here

Top 1000 SQL Performance Tips

Interactive session from MySQL Camp I:

Specific Query Performance Tips (see also database design tips for tips on indexes):

1. Use EXPLAIN to profile the query execution plan
2. Use Slow Query Log (always have it on!)
3. Don’t use DISTINCT when you have or could use GROUP BY
4. Insert performance
   1. Batch INSERT and REPLACE
   2. Use LOAD DATA instead of INSERT
5. LIMIT m,n may not be as fast as it sounds
6. Don’t use ORDER BY RAND() if you have > ~2K records
7. Use SQL_NO_CACHE when you are SELECTing frequently updated data or large sets of data
8. Avoid wildcards at the start of LIKE queries
9. Avoid correlated subqueries and in select and where clause (try to avoid in)
10. No calculated comparisons — isolate indexed columns
11. ORDER BY and LIMIT work best with equalities and covered indexes
12. Separate text/blobs from metadata, don’t put text/blobs in results if you don’t need them
13. Derived tables (subqueries in the FROM clause) can be useful for retrieving BLOBs without sorting them. (Self-join can speed up a query if 1st part finds the IDs and uses then to fetch the rest)
14. ALTER TABLE…ORDER BY can take data sorted chronologically and re-order it by a different field — this can make queries on that field run faster (maybe this goes in indexing?)
15. Know when to split a complex query and join smaller ones
16. Delete small amounts at a time if you can
17. Make similar queries consistent so cache is used
18. Have good SQL query standards
19. Don’t use deprecated features
20. Turning OR on multiple index fields (<5.0) into UNION may speed things up (with LIMIT), after 5.0 the index_merge should pick stuff up.
21. Don’t use COUNT * on Innodb tables for every search, do it a few times and/or summary tables, or if you need it for the total # of rows, use SQL_CALC_FOUND_ROWS and SELECT FOUND_ROWS()
22. Use INSERT … ON DUPLICATE KEY update (INSERT IGNORE) to avoid having to SELECT
23. use groupwise maximum instead of subqueries

Scaling Performance Tips:

1. Use benchmarking
2. isolate workloads don’t let administrative work interfere with customer performance. (ie backups)
3. Debugging sucks, testing rocks!
4. As your data grows, indexing may change (cardinality and selectivity change). Structuring may want to change. Make your schema as modular as your code. Make your code able to scale. Plan and embrace change, and get developers to do the same.

Network Performance Tips:

1. Minimize traffic by fetching only what you need.
   1. Paging/chunked data retrieval to limit
   2. Don’t use SELECT *
   3. Be wary of lots of small quick queries if a longer query can be more efficient
2. Use multi_query if appropriate to reduce round-trips

OS Performance Tips:

1. Use proper data partitions
   1. For Cluster. Start thinking about Cluster *before* you need them
2. Keep the database host as clean as possible. Do you really need a windowing system on that server?
3. Utilize the strengths of the OS
4. pare down cron scripts
5. create a test environment
6. source control schema and config files
7. for LVM innodb backups, restore to a different instance of MySQL so Innodb can roll forward
8. partition appropriately
9. partition your database when you have real data — do not assume you know your dataset until you have real data

MySQL Server Overall Tips:

1. innodb_flush_commit=0 can help slave lag
2. Optimize for data types, use consistent data types. Use PROCEDURE ANALYSE() to help determine the smallest data type for your needs.
3. use optimistic locking, not pessimistic locking. try to use shared lock, not exclusive lock. share mode vs. FOR UPDATE
4. if you can, compress text/blobs
5. compress static data
6. don’t back up static data as often
7. enable and increase the query and buffer caches if appropriate
8. config params — http://docs.cellblue.nl/2007/03/17/easy-mysql-performance-tweaks/ is a good reference
9. Config variables & tips:
   1. use one of the supplied config files
   2. key_buffer, unix cache (leave some RAM free), per-connection variables, innodb memory variables
   3. be aware of global vs. per-connection variables
   4. check SHOW STATUS and SHOW VARIABLES (GLOBAL|SESSION in 5.0 and up)
   5. be aware of swapping esp. with Linux, “swappiness” (bypass OS filecache for innodb data files, innodb_flush_method=O_DIRECT if possible (this is also OS specific))
   6. defragment tables, rebuild indexes, do table maintenance
   7. If you use innodb_flush_txn_commit=1, use a battery-backed hardware cache write controller
   8. more RAM is good so faster disk speed
   9. use 64-bit architectures
10. –skip-name-resolve
11. increase myisam_sort_buffer_size to optimize large inserts (this is a per-connection variable)
12. look up memory tuning parameter for on-insert caching
13. increase temp table size in a data warehousing environment (default is 32Mb) so it doesn’t write to disk (also constrained by max_heap_table_size, default 16Mb)
14. Run in SQL_MODE=STRICT to help identify warnings
15. /tmp dir on battery-backed write cache
16. consider battery-backed RAM for innodb logfiles
17. use –safe-updates for client
18. Redundant data is redundant

Storage Engine Performance Tips:

1. InnoDB ALWAYS keeps the primary key as part of each index, so do not make the primary key very large
2. Utilize different storage engines on master/slave ie, if you need fulltext indexing on a table.
3. BLACKHOLE engine and replication is much faster than FEDERATED tables for things like logs.
4. Know your storage engines and what performs best for your needs, know that different ones exist.
   1. ie, use MERGE tables ARCHIVE tables for logs
   2. Archive old data — don’t be a pack-rat! 2 common engines for this are ARCHIVE tables and MERGE tables
5. use row-level instead of table-level locking for OLTP workloads
6. try out a few schemas and storage engines in your test environment before picking one.

Database Design Performance Tips:

1. Design sane query schemas. don’t be afraid of table joins, often they are faster than denormalization
2. Don’t use boolean flags
3. Use Indexes
4. Don’t Index Everything
5. Do not duplicate indexes
6. Do not use large columns in indexes if the ratio of SELECTs:INSERTs is low.
7. be careful of redundant columns in an index or across indexes
8. Use a clever key and ORDER BY instead of MAX
9. Normalize first, and denormalize where appropriate.
10. Databases are not spreadsheets, even though Access really really looks like one. Then again, Access isn’t a real database
11. use INET_ATON and INET_NTOA for IP addresses, not char or varchar
12. make it a habit to REVERSE() email addresses, so you can easily search domains (this will help avoid wildcards at the start of LIKE queries if you want to find everyone whose e-mail is in a certain domain)
13. A NULL data type can take more room to store than NOT NULL
14. Choose appropriate character sets & collations — UTF16 will store each character in 2 bytes, whether it needs it or not, latin1 is faster than UTF8.
15. Use Triggers wisely
16. use min_rows and max_rows to specify approximate data size so space can be pre-allocated and reference points can be calculated.
17. Use HASH indexing for indexing across columns with similar data prefixes
18. Use myisam_pack_keys for int data
19. be able to change your schema without ruining functionality of your code
20. segregate tables/databases that benefit from different configuration variables

Other:

1. Hire a MySQL ™ Certified DBA
2. Know that there are many consulting companies out there that can help, as well as MySQL’s Professional Services.
3. Read and post to MySQL Planet at http://www.planetmysql.org
4. Attend the yearly MySQL Conference and Expo or other conferences with MySQL tracks (link to the conference here)
5. Support your local User Group (link to forge page w/user groups here)

 Authored by

Jay Pipes, Sheeri Kritzer, Bill Karwin, Ronald (”Jeremy Basher”) Bradford, Farhan “Frank Mash” Mashraqi, Taso Du Val, Ron Hu, Klinton Lee, Rick James, Alan Kasindorf, Eric Bergen, Kaj Arno, Joel Seligstein, Amy Lee