Another challenge to catch-out the unwary, is that the latest CentOS/RedHat YUM repository version of NGinx, the fantastically fast web server we use for Magento, has some case scenarios where even though it should be compressing CSS and Javascript, it simply doesn’t!

The reason why seems to be that most definitions for what types of files NGinx should compress posted across the forums of the web, include the “text/html” type, such as:

gzip_types text/plain text/html text/css application/json application/x-javascript text/xml application/javascript text/x-js;

Now the problem with this seems to be that NGinx throws a simple warning that it has already got “text/html” defined as it does this by default, however what it then doesn’t tell you is that it IGNORES all the other definitions that come after it in the same line.

So what that means is that if you have the line above in your config file, even though you are defining for example “text/css” as being a file type to compress, NGinx will ignore this as it stops reading the line as soon as it hits the “text/html” double definition.

To fix, remove “text/html” from your line (*and while you are there you might as well just define the types we are using) and it will all work again.

Here’s my line for reference:

gzip_types text/css application/x-javascript;

Magento upgrade time again, and as normal it should be really easy, but there are as expected a number of snags that come across the way so here’s the (pretty much) definitive way of how to do it and how to fix the problems/errors/snags that happen as you do!

1. Start with the official Magento upgrade guide at the link below

http://www.magentocommerce.com/wiki/1_-_installation_and_configuration/magento_1.5_to_1.6_upgrade

2. If when running your ./mage mage-setup you receive the following error message:

PHP Fatal error: Uncaught exception ‘Exception’ with message ‘Invalid login credentials’ in /path-to-your-website/downloader/lib/Mage/Connect/Ftp.php

then you have the incorrect ftp details for your site configured.

To fix this edit /path-to-your-website/downloader/connnect.cfg and change magento_root to your webserver/downloader directory and remote_config to you username/password and webserver root directory

3. If you initial page load after upgrade fails with the following error:

SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry during upgrade

Then you need to remove constraints from your database as per the advice here which for reference involves changing your app/etc/config.xml file so that the existing initstatements change from:

<initstatements>SET NAMES utf8</initstatements>

to:

<initstatements>SET NAMES utf8; SET FOREIGN_KEY_CHECKS=0; SET UNIQUE_CHECKS=0;</initstatements>

Another alternative is to remove the problematic line from the SQL upgrade script which is probably much neater [and we did both just in case]. Details of how to do that here and for reference you have to:

edit “app/code/core/Mage/Sales/sql/sales_setup/mysql4-upgrade-1.5.9.9-1.6.0.0.php” and comment out the command starting on line 753.

4. Final ‘gotcha’ is that your JS and CSS files can refuse to load, and instead start trying to use your path-to-your-website as their root path, rather than the / path that they should do.

This seems to be due to permissions on the /path-to-your-website/media/ directory which Magento wants to write to, so make sure that the web server has write access here (normally a recursive permission change to allow group writes should do the thing and for reference that is:)

chmod -R g+w media

5. Clear your cache (safe way to do this is to change directory into your var directory and remove *, else if you include a /var in your statement you’ve wiped half your operating system if your dangerously logged in as root or su’d up!!)

cd var
rm -rf cache/* session/*

6. Good to go

Now the final thing is to fix any bugs in your themes – we found some new fatals in the Navigation skin due to not checking for parent/children objects being null – so you may get a few of those too. You can fix those with a

and matching closing } for the if statement

7. More references

A good support thread for this is on the Magento boards here

EPiServer let’s you quite happily wrongly login, but then after the default amount of incorrect attempts puts you account into a locked status.

Sensible enough one would think, except that the error message on login, correct or now incorrect, never tells you this and instead just uses the same incorrect username and/or password message as before.

To correct this you need to reset your account status in the EPiServer database, and for reference it’s in the Authentication database in the aspnet_Membership table and stored in the column IsLockedOut.

To fix, simple reset the status of that field to 0 using a similar SQL query to the one below

UPDATE [Authentication].[dbo].[aspnet_Membership] SET [IsLockedOut] = 0 WHERE [LoweredEmail] = ‘[email protected]’;

Magento loves using lots of mySQL processes, so therefore mySQL needs to finely tuned to achieve this.

We ran some benchmarking tests today using mysqlreport as the benchmarking tool and as you can see from the results below, mySQL caching following some of the tips in the articles below can really make a difference

1. First test

As you can see with basic caching turned on and no massive tuning we get a cache insert to prune rate of 1.31:1 and no real cache hits

query_cache_size = 8M
table_cache = 128
innodb_buffer_pool_size = 8M

__ Query Cache _________________________________________________________
Memory usage 5.89M of 8.00M %Used: 73.58
Block Fragmnt 14.66%
Hits 1.21M 157.7/s
Inserts 389.92k 50.7/s
Insrt:Prune 1.31:1 11.9/s
Hit:Insert

2. Second test with recommendations from mysqltuner.pl and mysqlreport-3.5

Bigger cache better results

query_cache_size = 32M
table_cache = 512
innodb_buffer_pool_size = 256M

__ Query Cache _________________________________________________________
Memory usage 23.79M of 32.00M %Used: 74.33
Block Fragmnt 12.43%
Hits 3.50M 184.3/s
Inserts 681.81k 35.9/s
Insrt:Prune 2.55:1 21.8/s
Hit:Insert 5.13:1

Hit versus Insert of 5.13 to 1 – now that should make the world a much faster place

References:

http://www.mysqlperformanceblog.com/2007/11/01/innodb-performance-optimization-basics/

http://www.mysqlperformanceblog.com/2007/11/03/choosing-innodb_buffer_pool_size/

http://www.techiecorner.com/45/turn-on-mysql-query-cache-to-speed-up-mysql-query-performance/

http://webcache.googleusercontent.com/search?q=cache:wVQfPOsS3t0J:www.debianhelp.co.uk/mysqlperformance.htm+mysql+optomise&cd=10&hl=en&ct=clnk&gl=uk&client=firefox-a&source=www.google.co.uk

Inspired by the great presentations at the Magento Developers Paradise 2011 I’ve been following some of the code created there on GitHub and some of the presenters own projects and here’s 7 great Magento projects you should be following on GitHub:

Magento GitHub Sites

1. 2Boys1Shop / Twoboysoneshop_Configr

Multi-shop configurator shown at MDP 2011

https://github.com/2Boys1Shop/Twoboysoneshop_Configr

2. alistairstead / MageTool

Tool to quickly create now Mage extension frameworks

https://github.com/alistairstead/MageTool

3. dankocherga / MTool

Auto creates files for new Magento module

https://github.com/dankocherga/MTool

https://github.com/dankocherga/MTool/wiki/Manual

4. jirafe / magento-plugin

Magento plugin for Jirafe stats

https://github.com/jirafe/magento-plugin/wiki/Overview

5. firegento / firegento-dynamiccategory

Dynamic category

https://github.com/firegento/firegento-dynamiccategory

6. More Vanish options

http://www.kalenyuk.com.ua/magento-performance-optimization-with-varnish-cache-47.html

http://jirafeinc.tumblr.com/post/6392764424/varnish-caching-with-magento

7. Firegento

FireGento extension with core functionality for debugging Magento

https://github.com/firegento/firegento

The latest version of PHP 5.3 deprecates the ‘ereg’ function, which generates errors that break programs such as TimThumb, the automatic image thumbnail generator.

To fix this, replace the existing ‘ereg’ expressions with alternative functions as per below:

Replace:

if (ereg(‘http://’, $src) == true) {

With:

if (strpos (strtolower ($src), ‘http://’) !== false || strpos (strtolower ($src), ‘https://’) !== false) {

Replace:

if (ereg($site, $url_info['host']) == true) {

With:

if (strpos (strtolower ($url_info['host']), $site) !== false) {

and you are all good to go :->

References:

http://code.google.com/p/timthumb/

http://devthought.com/2009/06/09/fix-ereg-is-deprecated-errors-in-php-53/

Our client, Vanessa G, launched their first collection last night with a fantastic show at The Banqueting Hall, a former palace of James 1st no less!

We recorded the video live and you can see the fantastic clothes and event below

For us, this was a strange problem in FreeBSD where even with a vanilla latest version of WordPress the error still occurred and we got a white screen trying to login to our admin area.

It seems to be caused by FreeBSD incorrectly thinking that the required files have been included as asked for in PHP using a require_once function.

So to fix it we manually added the wp-includes/functions.php file and change the existing require_once for wp-load.php in ‘wp-admin/admin.php’ to be:

[modified wp-admin/admin.php lines 20 and new line 21]

require(dirname(dirname(__FILE__)) . ‘/wp-load.php’);
require(dirname(dirname(__FILE__)) . ‘/wp-includes/functions.php’);

[original for reference line 20]

require_once(dirname(dirname(__FILE__)) . ‘/wp-load.php’);

Problem solved!

Magento the nice Community Version available e-commerce platform that we are using at Skywire for a number of our client builds is incredibly feature rich, but with all of these features comes the trade-off that to get any kind of speed out of the system you either need SERIOUS server hardware, or an awful lot of performance tuning.

To be honest it can run like a real dog if you don’t really work at it!

Well we like to make things work hard at Skywire so went on a journey of discovery on how to make Magento fly, and here’s our understandings to share with everyone else.

1. Server software selection and tuning

Lots of articles out there about this around the web, but you can sum it up in a few points really.

- What webserver (Apache vs. Lighttpd vs. Nginx) and how many threads for that webserver you need. Interestingly, against every article out there, Apache 2 was faster for us that Lighttpd and NginX but I think this was to do with the PHP CGI access the other two were using being slower on our Media Temple server

- Fine tune your mySQL database – we found that the two great scripts mysqlreport and mysqltuner are your friends here

- Get rid of any other processes you don’t need that get in the way (xinetd, spam assassin etc.)

2. Turn on lots of Caching

Magento has caching so turn that on for starts, and then get a minify type plugin (there’s lots of them out there but ) to complement that and join all of your CSS and JS into a single compressed file.

Install a PHP Byte Code caching system to cache any code generated by PHP – we used XCache as it was available via yum but eAccellerator gets good reviews too [although it just hung in our environment].

3. Turn on the Page Compilation feature in Magento!

Yes, I know it’s labelled as Beta, and yes I know it falls over most of the time you run it, but if you run it from the command line, as the same user that owns your web files then it works just great creating a new single directory in /includes/src containing flattened files of all your Magento files with the naming format directory_directory_etc_filename.php

This shaved at least 1 second off of every page load for us (amazing but true) however was a job to install as it ignores any modules installed in /app/code/community.

No worries though, you can work around this by just copying the whole module directory to the /app/code/local directory and rerunning the compiler and then it works great.

4. Load you Magento cache directories into a memory filesystem

Sounds a strange thing to do but you can load your /var/cache/ directory into a memory based ‘tmpfs’ which makes it much faster. Also you can move your sessions to your database instead however this slowed things down for us so we left them as files.

Summary:

So once you’ve done this on a mid-spec Media Temple DV server you can reduce page times from about 10 seconds down to just over a second, which believe me seems fast compared to how clunky Magento can be when running. Have fun!

References:

Magento performance and optimization

http://www.magentocommerce.com/group/view/168/

How do I use the inbuilt magento profiler to see bottlenecks?

http://www.magentocommerce.com/group/blog/action/viewpost/1405/group/168/

Magento Compiler – Improve your performance

http://www.magentocommerce.com/group/blog/action/viewpost/1243/group/168/

9 Methods to Speed Up Magento – A Guide to Making Magento Faster

http://www.blastedthing.com/magento/questions/mag-9-methods-to-speed-up-magento-a-guide-to-making-magento-faster/

Magento performance hosting

http://yoast.com/magento-performance-hosting/

Magento Site Performance Optimization

http://www.yireo.com/tutorials/magento/magento-spo#compress-output-in-general

Performance is Key! – Notes on Magento’s Performance

http://www.magentocommerce.com/blog/comments/performance-is-key-notes-on-magentos-performance/

One of the recent reasons we were failing PCI compliance with some of our sites is because our forms that submit to themselves, namely those with code in the top and html in the bottom, use the PHP $REQUEST_URI to use their own URL as the action for the form to POST back to.

Now… the reason this is dangerous is because it includes all the query string parameters too so if you add code after the query string it includes it in the page = BAD

So…. how do we fix it?

The trick is to use the parse_url() function in PHP to ONLY grab the main path of the page itself rather than all the query strings.

So if we use

action="<?php echo parse_url($REQUEST_URI, PHP_URL_PATH);?>"

then we fix the security loop hole above and happy days

Additional: If we want to get the complete URL just without the query string we can tokenize the string using strtok as follows:

action="<php echo strtok($_SERVER['REQUEST_URI'],'?');?>"