Understanding ARC stats

[Jorge Gea]

Hi,

I'm working with the /proc/spl/kstat/zfs/arcstats of ZoL, and there are a
lot of things that I don't understand very well.

So, first question is: is there any documentation/manual where each of this
ARC values are well explained? I have looked for this at Oracle docs and
googling, but I haven't found anything accurate.

Now, some specific questions about it:
1. Looking only at this arcstats, is there a way to know if dedup tables
are fitting in RAM? I know how to calculate DDT sizes using zdb and zpool
status -D. But I would like to know a way to say if I have DDT at RAM or
not in an accurate way. I don't know what is the priority for the ARC (MRU,
MFU, DDTs, other metadata....)

2. Is there any way to know how much memory are beeing used by snapshots
and clones in an specific moment, also using this arcstats?

3. I have several systems with, theoretically, enaugh RAM to handle their
Deduplicated data. In general, 128Gb RAM for less than 9Tb of unique data.
I also have set the zfs_arc_max for using 120Gb and the zfs_arc_meta_limit
for using 60Gb. With this configuration systems should be happy, but in
some of them I have the arc_meta_used reaching the arc_meta_limit all the
time, So, sometimes the filesystem suffers performance issues. How could
this be explained?

Thanks for any help

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org

[richard.elling-icOkP4L6Z85BQ5RWgiF5GgC/]

Post by Jorge Gea
Hi,
I'm working with the /proc/spl/kstat/zfs/arcstats of ZoL, and there are a
lot of things that I don't understand very well.
So, first question is: is there any documentation/manual where each of
this ARC values are well explained? I have looked for this at Oracle docs
and googling, but I haven't found anything accurate.

Other than the source, which is well commented, the best posts are benr's
blogs
on http://www.cuddletech.com/blog

Post by Jorge Gea
1. Looking only at this arcstats, is there a way to know if dedup tables
are fitting in RAM?

No. The DDT is just part of metadata and it is not required to be in RAM.
The same goes
for all metadata... if you aren't using it, don't put in in ARC.

Post by Jorge Gea
I know how to calculate DDT sizes using zdb and zpool status -D. But I
would like to know a way to say if I have DDT at RAM or not in an accurate
way

The answer to this question won't help.

Post by Jorge Gea
. I don't know what is the priority for the ARC (MRU, MFU, DDTs, other
metadata....)

The ARC doesn't have priority. It does have MFU and MRU.

Post by Jorge Gea
2. Is there any way to know how much memory are beeing used by snapshots
and clones in an specific moment, also using this arcstats?

Memory? There is only demand and prefetch. If there is never demand or
prefetch then
the [meta]data won't be in ARC.

Post by Jorge Gea
3. I have several systems with, theoretically, enaugh RAM to handle their
Deduplicated data. In general, 128Gb RAM for less than 9Tb of unique data.
I also have set the zfs_arc_max for using 120Gb and the zfs_arc_meta_limit
for using 60Gb. With this configuration systems should be happy, but in
some of them I have the arc_meta_used reaching the arc_meta_limit all the
time, So, sometimes the filesystem suffers performance issues. How could
this be explained?

It think you just explained part of it. But it really depends on what you
are trying to do
and what performance you expect. Neither of these things are in arcstats.
-- richard

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org

[Jorge Gea]

Hi,

Thanks for the answer. I'll look at the source code then.

About DDT. Maybe i'm wrong, but if you dedup data and your DDTs are bigger
than your arc or l2arc, if you have it, you are going to suffer serious
performance issues. So best situation is to have it all at your primary
cache on RAM.

I'm trying to have this situation, all DDTs and metadata at RAM. But I'd
like to be able to monitor it, knowing if i really have all my DDTs at RAM
or if they are beeing accessed from the disk, to prevent performance
issues.

In general, what i want to do is to monitor and understand the memory
situation of a zfs system at any time, to decide if i need to increase the
resources or not. So is this possible looking only at arcstats? if not, are
there any other possible approach?

Thanks

Post by richard.elling-icOkP4L6Z85BQ5RWgiF5GgC/

Post by Jorge Gea
Hi,
I'm working with the /proc/spl/kstat/zfs/arcstats of ZoL, and there are a
lot of things that I don't understand very well.
So, first question is: is there any documentation/manual where each of
this ARC values are well explained? I have looked for this at Oracle docs
and googling, but I haven't found anything accurate.

Other than the source, which is well commented, the best posts are benr's
blogs
on http://www.cuddletech.com/blog

Post by Jorge Gea
1. Looking only at this arcstats, is there a way to know if dedup tables
are fitting in RAM?

No. The DDT is just part of metadata and it is not required to be in RAM.
The same goes
for all metadata... if you aren't using it, don't put in in ARC.

Post by Jorge Gea
I know how to calculate DDT sizes using zdb and zpool status -D. But I
would like to know a way to say if I have DDT at RAM or not in an accurate
way

The answer to this question won't help.

Post by Jorge Gea
. I don't know what is the priority for the ARC (MRU, MFU, DDTs, other
metadata....)

The ARC doesn't have priority. It does have MFU and MRU.

Post by Jorge Gea
2. Is there any way to know how much memory are beeing used by snapshots
and clones in an specific moment, also using this arcstats?

Memory? There is only demand and prefetch. If there is never demand or
prefetch then
the [meta]data won't be in ARC.

Post by Jorge Gea
3. I have several systems with, theoretically, enaugh RAM to handle their
Deduplicated data. In general, 128Gb RAM for less than 9Tb of unique data.
I also have set the zfs_arc_max for using 120Gb and the
zfs_arc_meta_limit for using 60Gb. With this configuration systems should
be happy, but in some of them I have the arc_meta_used reaching the
arc_meta_limit all the time, So, sometimes the filesystem suffers
performance issues. How could this be explained?

It think you just explained part of it. But it really depends on what you
are trying to do
and what performance you expect. Neither of these things are in arcstats.
-- richard
To unsubscribe from this group and stop receiving emails from it, send an

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org

[richard.elling-icOkP4L6Z85BQ5RWgiF5GgC/]

Post by Jorge Gea
Hi,
Thanks for the answer. I'll look at the source code then.
About DDT. Maybe i'm wrong, but if you dedup data and your DDTs are bigger
than your arc or l2arc, if you have it, you are going to suffer serious
performance issues. So best situation is to have it all at your primary
cache on RAM.

Any workload running in nearby cache is faster than a workload that isn't.
This true whether you run dedup or not.

A better way to approach this is to understand your working set size (WSS).
If you know your WSS, then you
can calculate or measure how much metadata goes with the data and thereby
arrive at the size of RAM needed
to keep everything in ARC.

Post by Jorge Gea
I'm trying to have this situation, all DDTs and metadata at RAM. But I'd
like to be able to monitor it, knowing if i really have all my DDTs at RAM
or if they are beeing accessed from the disk, to prevent performance issues.

The contents of ARC are constantly changing. Creating new blocks also
creates new metadata
and, if dedup is enabled, new DDT entries.

Post by Jorge Gea
In general, what i want to do is to monitor and understand the memory
situation of a zfs system at any time, to decide if i need to increase the
resources or not. So is this possible looking only at arcstats? if not, are
there any other possible approach?

arcstats only show the ARC contents at a high level. This is sufficient for
answering the questions
along the lines of "is my ARC too small?" "will an L2ARC help my workload?"
"is my L2ARC too
small?" "what is the ARC overhead for my L2ARC?" The answers change over
time as you workload
and WSS change. arcstats are a low-overhead way to track these changes over
time.
-- richard
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org

[Prakash Surya]

Hi,
I'm working with the /proc/spl/kstat/zfs/arcstats of ZoL, and there are a lot
of things that I don't understand very well.
So, first question is: is there any documentation/manual where each of this ARC
values are well explained? I have looked for this at Oracle docs and googling,
but I haven't found anything accurate.
1. Looking only at this arcstats, is there a way to know if dedup tables are
fitting in RAM? I know how to calculate DDT sizes using zdb and zpool status -
D. But I would like to know a way to say if I have DDT at RAM or not in an
accurate way. I don't know what is the priority for the ARC (MRU, MFU, DDTs,
other metadata....)

You can probably get most of what you're trying to answer by using the
dbufstat.py utility. We've exported information regarding what is
being cached at the dbuf layer, which that tool utilizes. You can see
exactly how much of an object (e.g. the ddt object in your case) is in
the dbuf hash table at any given time.

With that said, **beware**, there are some bugs with that tool that can
crash the system. On a lightly utilized system I've had no issues with
it, but I've hit issues running it on a heavily utilized system.

The script is contained in the ZoL source tree, just like arcstat.py.

2. Is there any way to know how much memory are beeing used by snapshots and
clones in an specific moment, also using this arcstats?

Nothing I know about. I've never tried, but dbufstat.py might help here
as long as the snapshots have different filesystem IDs (which I think
they do).

3. I have several systems with, theoretically, enaugh RAM to handle their
Deduplicated data. In general, 128Gb RAM for less than 9Tb of unique data.
I also have set the zfs_arc_max for using 120Gb and the zfs_arc_meta_limit for
using 60Gb. With this configuration systems should be happy, but in some of
them I have the arc_meta_used reaching the arc_meta_limit all the time, So,
sometimes the filesystem suffers performance issues. How could this be
explained?

No easy answer here. What version of ZoL are you using? I'd need to look
at a snapshot of arcstats when the system is in a "bad" state to provide
any useful commentary.

[Jorge Gea]

Hi,

Very interesting tool, dbufstat.py. Even if it has some bugs, it shows me
clearly where I have to look for the information that I need. I didn't know
this tool, because right know I'm using ZoL 0.6.1 and 0.6.2 where the tool
was not included yet. It seems that the /proc/spl/kstat/zfs/dbufs is
needed for this and it's not present at this versions. But I have plans to
upgrade my systems soon, so thank you very much!

Regarding the performance issues, I have been doing some tests. I think I'm
having problems when zfs has to release memory and I'm trying with greater
values for parameters zfs_arc_shrink_shift and zfs_arc_meta_prune. Right
now, the problem has not showed itself again. But I'm still not sure if
it's completely solved, I need more time. If it happens again, I'll capture
the arcstats snapshot and paste it here.

Thanks for the answers

Post by Jorge Gea

Post by Jorge Gea
Hi,
I'm working with the /proc/spl/kstat/zfs/arcstats of ZoL, and there are

a lot

Post by Jorge Gea
of things that I don't understand very well.
So, first question is: is there any documentation/manual where each of

this ARC

Post by Jorge Gea
values are well explained? I have looked for this at Oracle docs and

googling,

Post by Jorge Gea
but I haven't found anything accurate.
1. Looking only at this arcstats, is there a way to know if dedup tables

are

Post by Jorge Gea
fitting in RAM? I know how to calculate DDT sizes using zdb and zpool

status -

Post by Jorge Gea
D. But I would like to know a way to say if I have DDT at RAM or not in

an

Post by Jorge Gea
accurate way. I don't know what is the priority for the ARC (MRU, MFU,

DDTs,

Post by Jorge Gea
other metadata....)

You can probably get most of what you're trying to answer by using the
dbufstat.py utility. We've exported information regarding what is
being cached at the dbuf layer, which that tool utilizes. You can see
exactly how much of an object (e.g. the ddt object in your case) is in
the dbuf hash table at any given time.
With that said, **beware**, there are some bugs with that tool that can
crash the system. On a lightly utilized system I've had no issues with
it, but I've hit issues running it on a heavily utilized system.
The script is contained in the ZoL source tree, just like arcstat.py.

Post by Jorge Gea
2. Is there any way to know how much memory are beeing used by snapshots

and

Post by Jorge Gea
clones in an specific moment, also using this arcstats?

Nothing I know about. I've never tried, but dbufstat.py might help here
as long as the snapshots have different filesystem IDs (which I think
they do).

Post by Jorge Gea
3. I have several systems with, theoretically, enaugh RAM to handle their
Deduplicated data. In general, 128Gb RAM for less than 9Tb of unique

data.

Post by Jorge Gea
I also have set the zfs_arc_max for using 120Gb and the

zfs_arc_meta_limit for

Post by Jorge Gea
using 60Gb. With this configuration systems should be happy, but in some

of

Post by Jorge Gea
them I have the arc_meta_used reaching the arc_meta_limit all the time,

So,

Post by Jorge Gea
sometimes the filesystem suffers performance issues. How could this be
explained?

No easy answer here. What version of ZoL are you using? I'd need to look
at a snapshot of arcstats when the system is in a "bad" state to provide
any useful commentary.
--
Cheers, Prakash

Post by Jorge Gea
Thanks for any help
To unsubscribe from this group and stop receiving emails from it, send

an email
To unsubscribe from this group and stop receiving emails from it, send an

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org