From simon at technocool.net  Mon Jan 25 23:55:00 2010
From: simon at technocool.net (Simon Waters)
Date: Tue, 26 Jan 2010 07:55:00 +0000
Subject: [DCPM] Profiling
Message-ID: <4B5E9FD4.300@technocool.net>

Having fun profiling our big Perl app at work.

I think I have NYTProf working sensibly.

A lot of time is spent in YAML, and attempt to switch to YAML::XS seemed
subjectively faster, but generated a few issues and we never rolled the
change out. The issues are largely I suspect we are using YAML wrongly
but consistently wrongly.

However when I switch YAML::XS for YAML the profiling figures still
claim the method that takes most time in YAML is taking a long time, but
it isn't assigning this time to YAML::XS.

Is this typical?

The process is running as an Apache2 fcgid server if this is relevant?
It sure made figuring out how to run NYTProf interesting, although of
course once I understood what change to make it was only a four line
change to the Apache virtualhost section.

 Simon

From aaron.trevena at gmail.com  Wed Jan 27 07:26:41 2010
From: aaron.trevena at gmail.com (Aaron Trevena)
Date: Wed, 27 Jan 2010 15:26:41 +0000
Subject: [DCPM] Profiling
In-Reply-To: <4B5E9FD4.300@technocool.net>
References: <4B5E9FD4.300@technocool.net>
Message-ID: <edcafa7d1001270726m7eb27d24v8ff7660422249b22@mail.gmail.com>

2010/1/26 Simon Waters <simon at technocool.net>:
> Having fun profiling our big Perl app at work.
>
> I think I have NYTProf working sensibly.
>
> A lot of time is spent in YAML, and attempt to switch to YAML::XS seemed
> subjectively faster, but generated a few issues and we never rolled the
> change out. The issues are largely I suspect we are using YAML wrongly
> but consistently wrongly.
>
> However when I switch YAML::XS for YAML the profiling figures still
> claim the method that takes most time in YAML is taking a long time, but
> it isn't assigning this time to YAML::XS.
>
> Is this typical?

It could be including the xs in the lowest level perl call ?

Are you using version 3 ? ISTR it provided improved XS handling.

A.


-- 
Aaron J Trevena, BSc Hons
http://www.aarontrevena.co.uk
LAMP System Integration, Development and Consulting

From simon at technocool.net  Wed Jan 27 09:03:57 2010
From: simon at technocool.net (Simon Waters)
Date: Wed, 27 Jan 2010 17:03:57 +0000
Subject: [DCPM] Profiling
In-Reply-To: <edcafa7d1001270726m7eb27d24v8ff7660422249b22@mail.gmail.com>
References: <4B5E9FD4.300@technocool.net>
	<edcafa7d1001270726m7eb27d24v8ff7660422249b22@mail.gmail.com>
Message-ID: <4B6071FD.7050203@technocool.net>

Aaron Trevena wrote:
>
>> However when I switch YAML::XS for YAML the profiling figures still
>> claim the method that takes most time in YAML is taking a long time, but
>> it isn't assigning this time to YAML::XS.
>>
>> Is this typical?
> 
> It could be including the xs in the lowest level perl call ?

That is what I thought, but as far as I can see NYTProf is assiging it
to our routine "read_file", but not to any line in "read_file" (when
presumably YAML::XS::LoadFile is consuming most of the time - since it
doesn't do much else).

> Are you using version 3 ? ISTR it provided improved XS handling.

Version 3 of NYTProf - yes.

It seems to assign time as I would expect for most calls to regular
operations(?) that are implemented in C, just not here.

I'm fairly confident the YAML parsing time is a significant issue, but
aside from spending too long parsing it, we read the files too often,
they are bigger than they need to be, and as I mentioned we don't use
"LoadFile" and "DumpFile" (rather someone calls Load and Dump on data
loaded or written to files, which is almost but not exactly the same
thing - think UTF8 encoding).

The module YAML (not YAML::XS) has a regular expression at the heart of
it burning the CPU (with a comment to that effect next to it), is
written to a prerelease spec of YAML, and I think is probably best
avoided (says the guy with millions of such files to manage - ho hum).