Discussion:
[xs-devel] Switchless shared iSCSI Network with /30 links
Vinícius Ferrão
2015-02-14 21:26:53 UTC
Permalink
Hello guys,

I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.

At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.

I’ve looked for help on ServerFault and FreeNAS Forums, since I’m running FreeNAS as iSCSI service. The discussion is going on those links:

http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch <http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch>
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/ <https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/>

It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.

If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.

Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)

Thanks in advance,
Vinicius.
Tobias Kreidl
2015-02-14 22:14:20 UTC
Permalink
Frankly speaking, it doesn't seem that it's a good idea to try to try to
squeeze something out of a technology that is not built-in or
future-safe. Would you want to try to connect a fiber-channel storage
device without a fiber channel switch (a very similar situation)? There
are other alternatives, for example, to connect iSCSi storage to
another, XenServer-independent server and then serve storage from it via
NFS or use NFS in the first place and not make use at all of iSCSI
technology. NFS has come a long way and in many cases, can be equivalent
in I/O performance to iSCSI.

The intercommunications and ability to share SRs within a pool are a
significant consideration and cannot be ignored as part of the storage
support protocols within XenServer. XenServer provides a number of
options for storage connectivity to cover a wide range of needs,
preferences and costs. Often, trying to save money in the wrong area
results in more headaches later on.

That all being said, it /is/ possible to bypass the SR mechanism
altogether (obviously, not a supported configuration) and connect at
least Linux VMs directly to iSCSI storage using open iSCSI, but I have
not explored this without still involving a network switch. As you
indicated, it's also not clear if XenServer will remain 100% open iSCSI
compatible or not in the future. It also takes aways some of the things
an SR can offer, like VM cloning, storage migration, etc.

Finally, I am not sure I see where the lack of a switch factors in in
improving performance. Networks switches are much more capable and much
faster than any storage device or host in routing packets. The amount of
overhead they add is tiny.

Regards,
-=Tobias
Post by Vinícius Ferrão
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using
a iSCSI SR without a Switch. The main goal is to reduce the TCO of the
solution, get some extra performance removing the overhead that a
managed switch puts on the network and reduce the complexity of the
network.
At a first glance I thought that I will only achieve those kind of
setup with a distributed switch, so DVSC should be an option. But as
Tim said it’s only a interface to control some policies.
I’ve looked for help on ServerFault and FreeNAS Forums, since I’m
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/
It appears that XenServer cannot support switchless iSCSI storages
without a switch, because XS needs the same network block on all the
XS hosts to create a shared storage between the hosts. So the ideia of
point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate
PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR
creation and try to create an SR with all the PBD’s from the XS hosts.
But I was unable to test this setting due to lack of my knowledge when
trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the
developers, to see if this method of building the iSCSI network is
viable, since as Tim said, XS is not 100% open-iscsi upstream and if
it could be integrated on next versions of XenServer via the XenCenter
GUI. I think that’s a very common architecture and VMware is doing
this on small pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Vinícius Ferrão
2015-02-15 05:44:11 UTC
Permalink
Hello Tobias,
Thanks for the reply, even to discourage me :)

I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.

Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.

Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked
 Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.

Leaving all those “political” points aside, let’s talk technically :)

I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: Loading Image... <Loading Image...

I’m really excited because it worked. So let’s me explain what I’ve did.

First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.

Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.

That’s when things got interesting. To create the PBD, I do need the iSCSI working in the host, and consequently the multipath connection. So I done the process in the terminal with the following commands:

#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1

#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login

#Enable Multipath
multipath
multipath -ll

After all I was able to create the PBD with those new settings. The tricky part was to pass the required “device-config” values via the xe pbd-create command. I wasn’t aware of the method, but after 4 tries I understood the process and typed the following command:

#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260

Basically everything necessary is in there. The IP’s, the name, the multipath link, the LUN values, everything. After this, I just plugged the PDB via the CLI:

#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342

And BOOM! It worked. As you can see in the screenshot.

To prove that’s everything is working fine, I’ve created a new VM, installed FreeBSD 10 on it, done a pkg install xe-guest-utilities and requested a XenMotion operation. Oh boy, it worked:

Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms

Pics: Loading Image... <Loading Image...

I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.

What I ask now: a feedback, if possible.

Thanks in advance,
Vinícius.

PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Post by Vinícius Ferrão
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch <http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch>
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/ <https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/>
It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Tobias Kreidl
2015-02-15 07:25:54 UTC
Permalink
Hi, Vinícius:
First off, no apologies needed for any language issues -- if I spoke
Brazilian Portuguese as well, I'd be very proud of myself! :-) Second,
your persistence certainly is commendable so don't necessarily think I
am being overly discouraging! :-) This was interesting enough that I
also stayed up late to think about it some more.

I take it what you are saying is that you have in essence created a
separate PBD for each host that points to the same SR, is that right?
That is what in the admin guide falls under the chapter "Introducing an
SR" (In XS 6.5, for example, chapter 5.7.2) and indeed, as it states,
this needs to be done separately for each host as each needs to plug in
the PBD corresponding to that same SR. So, I do see what you are saying
as to why this currently does not work via XenCenter for connections
that do not go through a network switch. In the CLI-based procedure,
you save one step on each host by cleverly using the
device-config:multihomelist and decice-config:multiSession options.

It, however, might still work (sort of) via XenCenter, so let me ask
this: If you do this operation from a single host, do you see the SR
show up as broken but at least visible in XenCenter? So on one host,
issue your pbd-create something like:

xe pbd-create type=lvmoiscsi host-uuid=/UUID-of-host/// sr-uuid=/UUID-of-SR/// \
device-config:target=192.168.20.1 \
device-config:targetIQN=192.168.20.13:... \ (or "*" if unambiguous)
device-config:SCSIid=1FreeBSD_iSCSI_Disk_002...


(for a target IQN of "*" see, for example: http://lists.xenproject.org/archives/html/xen-api/2011-05/msg00023.html)

If that works for the one host and the SR shows up as a /broken device/
(in red) in XenCenter, there is a good chance that all you might need to
do is run the /SR repair function/ from XenCenter to get this fixed on
_all_ the hosts.

Another question is if you reboot one XenServer host in the pool, does
the PBD then get automatically plugged back in? Also, what does
"multipath -ll" display now on each host? What happens if you try to
move a VDI to a different SR and back again? You'd then also need to
make sure the SR could be properly forgotten, detached, reattached,
etc. If all that looks and behaves like a standard SR, then I think you
may indeed have a winning combination! It may take some additional work
to stabilize things under all general circumstances, of course.

Best regards,
-=Tobias
Post by Vinícius Ferrão
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it, they
recommend, and they even sell a solution using DELL hardware with two
machines and a shared storage running with 10GbE links. Point-ot-point
networks, dual controllers on the storage side and two different
network for iSCSI. They sell it at least here in Brazil. What I’m
trying to do is achieve the same topology with XenServer and commodity
hardware.
Why I want to do this? Because I do believe that XenServer is a better
product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different markets.
For example, I’ve used Fibre Channel only one time in my life. And,
believe it or not, it was a switchless configuration. And it worked

Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS
software. I do have the pictures of the topology, but I need to search
for it. At that time, I wasn’t the guy who created the solution, but I
understood that it was a huge money saving, because the FC switches
are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was
necessary in the CLI, but I was comfortable with this. The following
https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0
I’m really excited because it worked. So let’s me explain what I’ve did.
First I started creating the SR via XenCenter, it failed as expected.
But to bypass I just created the SR via XenCenter with the two Storage
IP’s of the Pool Master. So the process went OK, but it started
broken, since the second XS host cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR UUID
created by XenCenter and the referenced PBDs. According to what I know
about XS, the PBD’s are the physical connect from separate XS hosts to
the shared storage. So it’s not related to other hosts, it’s
exclusively to the host machine that I’m using. Understood that, the
ideia is to destroy the created PBD on the second XS host and recreate
it with the proper settings and IP addresses.
That’s when things got interesting. To create the PBD, I do need the
iSCSI working in the host, and consequently the multipath connection.
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260
--login
iscsiadm -m node --targetname
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260
--login
#Enable Multipath
multipath
multipath -ll
After all I was able to create the PBD with those new settings. The
tricky part was to pass the required “device-config” values via the xe
pbd-create command. I wasn’t aware of the method, but after 4 tries I
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f
host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|"
device-config:target=192.168.20.1,192.168.21.1
device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260
device-config:targetIQN=*
device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000
device-config:port=3260
Basically everything necessary is in there. The IP’s, the name, the
multipath link, the LUN values, everything. After this, I just plugged
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
To prove that’s everything is working fine, I’ve created a new VM,
installed FreeBSD 10 on it, done a pkg install xe-guest-utilities and
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0
I thinks this is sufficient for now, to prove that the concept works
and XenServer can create this kind of topology. It’s just not
available on the GUI. If XenCenter supports this kind of SR on the
next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was really
anxious to report this in the list.
Post by Tobias Kreidl
Frankly speaking, it doesn't seem that it's a good idea to try to try
to squeeze something out of a technology that is not built-in or
future-safe. Would you want to try to connect a fiber-channel storage
device without a fiber channel switch (a very similar situation)?
There are other alternatives, for example, to connect iSCSi storage
to another, XenServer-independent server and then serve storage from
it via NFS or use NFS in the first place and not make use at all of
iSCSI technology. NFS has come a long way and in many cases, can be
equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a
significant consideration and cannot be ignored as part of the
storage support protocols within XenServer. XenServer provides a
number of options for storage connectivity to cover a wide range of
needs, preferences and costs. Often, trying to save money in the
wrong area results in more headaches later on.
That all being said, it /is/ possible to bypass the SR mechanism
altogether (obviously, not a supported configuration) and connect at
least Linux VMs directly to iSCSI storage using open iSCSI, but I
have not explored this without still involving a network switch. As
you indicated, it's also not clear if XenServer will remain 100% open
iSCSI compatible or not in the future. It also takes aways some of
the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in
improving performance. Networks switches are much more capable and
much faster than any storage device or host in routing packets. The
amount of overhead they add is tiny.
Regards,
-=Tobias
Post by Vinícius Ferrão
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of
using a iSCSI SR without a Switch. The main goal is to reduce the
TCO of the solution, get some extra performance removing the
overhead that a managed switch puts on the network and reduce the
complexity of the network.
At a first glance I thought that I will only achieve those kind of
setup with a distributed switch, so DVSC should be an option. But as
Tim said it’s only a interface to control some policies.
I’ve looked for help on ServerFault and FreeNAS Forums, since I’m
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/
It appears that XenServer cannot support switchless iSCSI storages
without a switch, because XS needs the same network block on all the
XS hosts to create a shared storage between the hosts. So the ideia
of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate
PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on
SR creation and try to create an SR with all the PBD’s from the XS
hosts. But I was unable to test this setting due to lack of my
knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the
developers, to see if this method of building the iSCSI network is
viable, since as Tim said, XS is not 100% open-iscsi upstream and if
it could be integrated on next versions of XenServer via the
XenCenter GUI. I think that’s a very common architecture and VMware
is doing this on small pools! So let’s do it equally and of course
surpass them. :)
Thanks in advance,
Vinicius.
Vinícius Ferrão
2015-02-16 20:34:26 UTC
Permalink
Hello Tobias,
Thanks for the support :)

I will separated this message in topics to facilitate the reading.

===============================

Yes in the essence I've created separated PBD's for each XS host. I just skipped the process for the first host because I created, as you said, a broken device connection.

To comply with some consistent checks, as you said, I've done the following:

1. Unplugged the PBD of the second XS host. The one that I've created the PBD via ter terminal.
xe pbd-unplug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342 (this is the UUID of the second host)

Immediately XenCenter reported a broken iSCSI connection: Loading Image... <Loading Image...

After I issued a SR repair request on XenCenter:
Loading Image... <Loading Image...

And finally XenCenter issued the repair procedure without any problem.

===============================

About reboots on the hosts. I've tried to reboot only the second machine and it returned without any problem, even with the iSCSI SR.

A second test was made, rebooting the pool master and the second machine. Both returned without any problem, with the shared SR working.

The only problem was a unrelated one, sometimes my Pool Master simples waits for some input on Syslinux. I need to press enter to continue booting. I don't know why this is happening. But I will find out later.

So I think it's pretty stable on reboots.

===============================

About the multipath -ll command. On the first host it reports:

[***@xenserver1 ~]# multipath -ll
1FreeBSD_iSCSI_Disk_002590946b34000 dm-0 FreeBSD,iSCSI Disk
size=4.0T features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 8:0:0:0 sdb 8:16 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 9:0:0:0 sdc 8:32 active ready running

On the second one:

[***@xenserver2 ~]# multipath -ll
1FreeBSD_iSCSI_Disk_002590946b34000 dm-0 FreeBSD,iSCSI Disk
size=4.0T features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 8:0:0:0 sdb 8:16 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 9:0:0:0 sdc 8:32 active ready running

As for I know, everything appears to be fine.

===============================

Finally about the basic SR operations:

1. Detach [OK]
Loading Image... <Loading Image...

2. Reattach (with the Repair option in XenCenter) [OK]
Loading Image... <Loading Image...

3. Forget [OK]
It has gone without problem. Remover the SR and even the modified PBD's. (Checked through xe-pbd list)

So I think SR is pretty stable.

What's missing now is the ability to create this topology on XenCenter. It appears to be technically supported to do this in XenServer, the only missing factor is the ability to create this with the XenCenter wizard.

===============================

Now Tobias, what I want to know is what's the meaning of some specific device-config options when creating the PBD manually:

multiSession:
multihomelist:
targetIQN: when to use * and when to use IP addresses.

Well, I think this is everything.

Thanks in advance,
Vinicius.
First off, no apologies needed for any language issues -- if I spoke Brazilian Portuguese as well, I'd be very proud of myself! :-) Second, your persistence certainly is commendable so don't necessarily think I am being overly discouraging! :-) This was interesting enough that I also stayed up late to think about it some more.
I take it what you are saying is that you have in essence created a separate PBD for each host that points to the same SR, is that right? That is what in the admin guide falls under the chapter "Introducing an SR" (In XS 6.5, for example, chapter 5.7.2) and indeed, as it states, this needs to be done separately for each host as each needs to plug in the PBD corresponding to that same SR. So, I do see what you are saying as to why this currently does not work via XenCenter for connections that do not go through a network switch. In the CLI-based procedure, you save one step on each host by cleverly using the device-config:multihomelist and decice-config:multiSession options.
xe pbd-create type=lvmoiscsi host-uuid=UUID-of-host sr-uuid=UUID-of-SR \
device-config:target=192.168.20.1 \
device-config:targetIQN=192.168.20.13:... \ (or "*" if unambiguous)
device-config:SCSIid=1FreeBSD_iSCSI_Disk_002...
(for a target IQN of "*" see, for example: http://lists.xenproject.org/archives/html/xen-api/2011-05/msg00023.html)
<>If that works for the one host and the SR shows up as a broken device (in red) in XenCenter, there is a good chance that all you might need to do is run the SR repair function from XenCenter to get this fixed on all the hosts.
Another question is if you reboot one XenServer host in the pool, does the PBD then get automatically plugged back in? Also, what does "multipath -ll" display now on each host? What happens if you try to move a VDI to a different SR and back again? You'd then also need to make sure the SR could be properly forgotten, detached, reattached, etc. If all that looks and behaves like a standard SR, then I think you may indeed have a winning combination! It may take some additional work to stabilize things under all general circumstances, of course.
Best regards,
-=Tobias
Post by Vinícius Ferrão
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked
 Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0 <https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0>
I’m really excited because it worked. So let’s me explain what I’ve did.
First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0 <https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0>
I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Post by Vinícius Ferrão
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch <http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch>
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/ <https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/>
It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Dave Scott
2015-02-15 14:22:16 UTC
Permalink
Hi,
Post by Vinícius Ferrão
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked… Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0
I’m really excited because it worked. So let’s me explain what I’ve did.
First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I’d like to understand your configuration better— have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?

Thanks,
Dave
Post by Vinícius Ferrão
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0
I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Post by Vinícius Ferrão
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/
It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Vinícius Ferrão
2015-02-16 20:38:59 UTC
Permalink
Hello Dave,

In total I have four paths. Two in each host. I've made a drawning using ASCII characters, hope this can clarify the architecture:

+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+

The Storage has four gigabit ethernet adapters with the following IP addresses:

192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30

On the first XenServer there are two interfaces with those IP's:

192.168.10.2/30
192.168.11.2/30

And on the second host the equivalent configuration:

192.168.20.2/30
192.168.21.2/30

That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.

To exemplify a little more heres a screenshot of the network configuration made on XenCenter:
Loading Image...

Cheers,
Vinícius.
Post by Dave Scott
Hi,
Post by Vinícius Ferrão
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked… Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0
I’m really excited because it worked. So let’s me explain what I’ve did.
First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I’d like to understand your configuration better— have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?
Thanks,
Dave
Post by Vinícius Ferrão
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0
I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Post by Vinícius Ferrão
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/
It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Tobias Kreidl
2015-02-16 20:53:48 UTC
Permalink
No reason to necessarily use four separate subnets. You could have
192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and 2.2
on the other controller (on, for example, an MD36xx that is standard).
If however it is something like an MD32xx which recommends four separate
subets, then of course, use four. One should follow the vendor's
specifications.

I will respond in more detail later when I get a chancee, Vinícius, but
after a quick read, I agree with your findings from your other email. It
would appear that the initial connection is indeed evidently a missing
function in XenCenter and perhaps all that is needed to make this work
"out of the box". As to the wildcard discovery, it's generally preferred
as it allows the host to figure out the path on its own. The only reason
I could see not using a wildcard discovery would be if there were any
chance for overlaps and ambiguous connections.

Regards,
--Tobias
Post by Vinícius Ferrão
Hello Dave,
+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+
192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30
192.168.10.2/30
192.168.11.2/30
192.168.20.2/30
192.168.21.2/30
That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0
Cheers,
Vinícius.
Post by Dave Scott
Hi,
Post by Vinícius Ferrão
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked… Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0
I’m really excited because it worked. So let’s me explain what I’ve did.
First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I’d like to understand your configuration better— have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?
Thanks,
Dave
Post by Vinícius Ferrão
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0
I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Post by Vinícius Ferrão
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/
It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Vinícius Ferrão
2015-02-16 21:20:55 UTC
Permalink
Hi Tobias,

As for I know, the Dell MD36xx is a dual controller based storage. So it's basically "two computers". Considering this with the added knowledge that I've about TCP networking there's a golden rule that says: only one IP address on a given subnet on a single computer. So I can't use the same network on my case. That's why I use point-to-point links too.

Since I've only one machine acting as a storage server (It's a Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do need four separate networks. FreeNAS is FreeBSD based, and BSD enforces this rule by default. I can't have two IP addresses of the same subnet on same machine, even if I manually configure it.

If this wasn't bad TCP network practice, I would be able to configure the SR via XenCenter, since it will "find" the same network on the others hosts without problem.

About the XenCenter, it really appears to be only a missing function on the software. If I was a good programmer I would try to implement this. But this is beyond my knowledge... :(

Thanks once again,
Vinícius.
No reason to necessarily use four separate subnets. You could have 192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and 2.2 on the other controller (on, for example, an MD36xx that is standard). If however it is something like an MD32xx which recommends four separate subets, then of course, use four. One should follow the vendor's specifications.
I will respond in more detail later when I get a chancee, Vinícius, but after a quick read, I agree with your findings from your other email. It would appear that the initial connection is indeed evidently a missing function in XenCenter and perhaps all that is needed to make this work "out of the box". As to the wildcard discovery, it's generally preferred as it allows the host to figure out the path on its own. The only reason I could see not using a wildcard discovery would be if there were any chance for overlaps and ambiguous connections.
Regards,
--Tobias
Post by Vinícius Ferrão
Hello Dave,
+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+
192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30
192.168.10.2/30
192.168.11.2/30
192.168.20.2/30
192.168.21.2/30
That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0
Cheers,
Vinícius.
Post by Dave Scott
Hi,
Post by Vinícius Ferrão
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked… Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0
I’m really excited because it worked. So let’s me explain what I’ve did.
First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I’d like to understand your configuration better— have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?
Thanks,
Dave
Post by Vinícius Ferrão
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0
I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Post by Vinícius Ferrão
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/
It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Tobias Kreidl
2015-02-16 23:02:16 UTC
Permalink
Yes, correct. The MD36xx looks like two separate controller units, each
acting as a separate device. For a standalone server or some storage
devices, that's right that you need a separate subnet for each connection.

Because a single connection isn't seeing the broadcast from the other
connectors at the time of the initial connection from one host, the pool
is unaware of the others. The "repair" operation causes this to be
conducted from each host, and as I guessed, then fixes the SR
connectivity for the entire pool. I am speculating that the request for
a new connection of this type via XenCenter could perhaps be solved by
propagating that initial connection process to the rest of the hosts in
the pool and then rescan the SR.

-=Tobias
Post by Vinícius Ferrão
Hi Tobias,
As for I know, the Dell MD36xx is a dual controller based storage. So it's basically "two computers". Considering this with the added knowledge that I've about TCP networking there's a golden rule that says: only one IP address on a given subnet on a single computer. So I can't use the same network on my case. That's why I use point-to-point links too.
Since I've only one machine acting as a storage server (It's a Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do need four separate networks. FreeNAS is FreeBSD based, and BSD enforces this rule by default. I can't have two IP addresses of the same subnet on same machine, even if I manually configure it.
If this wasn't bad TCP network practice, I would be able to configure the SR via XenCenter, since it will "find" the same network on the others hosts without problem.
About the XenCenter, it really appears to be only a missing function on the software. If I was a good programmer I would try to implement this. But this is beyond my knowledge... :(
Thanks once again,
Vinícius.
No reason to necessarily use four separate subnets. You could have 192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and 2.2 on the other controller (on, for example, an MD36xx that is standard). If however it is something like an MD32xx which recommends four separate subets, then of course, use four. One should follow the vendor's specifications.
I will respond in more detail later when I get a chance, Vinícius, but after a quick read, I agree with your findings from your other email. It would appear that the initial connection is indeed evidently a missing function in XenCenter and perhaps all that is needed to make this work "out of the box". As to the wildcard discovery, it's generally preferred as it allows the host to figure out the path on its own. The only reason I could see not using a wildcard discovery would be if there were any chance for overlaps and ambiguous connections.
Regards,
--Tobias
Post by Vinícius Ferrão
Hello Dave,
+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+
192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30
192.168.10.2/30
192.168.11.2/30
192.168.20.2/30
192.168.21.2/30
That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0
Cheers,
Vinícius.
Post by Dave Scott
Hi,
Post by Vinícius Ferrão
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked… Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0
I’m really excited because it worked. So let’s me explain what I’ve did.
First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I’d like to understand your configuration better— have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?
Thanks,
Dave
Post by Vinícius Ferrão
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0
I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Post by Vinícius Ferrão
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/
It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Vinícius Ferrão
2015-02-17 00:24:02 UTC
Permalink
Tobias,

I've done more tests about the SR creation over XenCenter.

All the process definitely occurs only on the pool master. Even if the specified networks are not part of the networks of the host. The connection is tried on the default route.

I requested a new Software iSCSI SR with the following address:
192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1

The target IQN is discovered without any problem, since the iSCSI Storage reports the LUNs on any interface.

But when trying to discover the LUN, the process only is done on the pool master. During the discovery process, I left a watch command on both XS to with "netstat -an | grep 192.168" and the results are:

On the pool master, there's a lot of things going on:
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 192.168.10.2:55870 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52654 192.168.11.1:3260 TIME_WAIT
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.11.2:52656 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*

During the discovery, look at the 192.168.21.1 address being searched through the default route:
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
******************
tcp 0 1 10.7.0.101:56481 192.168.21.1:3260 SYN_SENT
******************
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*

Even addresses that does not exist on the pool (since the IQN discovery process reports them):
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
******************
tcp 0 1 10.7.0.101:52787 192.168.30.1:3260 SYN_SENT
******************
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*

If you're wondering the 192.168.30.1 is reported because in the future we will add another XS host to the pool. The point here is that XenCenter should not lookup for this network since it isn't specified during the Software iSCSI creation. On my understanding it should only lookup for the specified addresses.

And to finish the tests, the second XS host remains equally during all the SR creation phase:
[***@xenserver2 ~]# netstat -an | grep -i 192.168
tcp 0 48 10.7.0.102:22 192.168.172.1:58046 ESTABLISHED
udp 0 0 192.168.21.2:123 0.0.0.0:*
udp 0 0 192.168.20.2:123 0.0.0.0:*

As you said, perhaps propagating the initial connection through the host will solve the problem without modifying the XenCenter interface. A more sophisticated approach would be query the network-list over XAPI, retrieve the PIFs associated to those networks and them just check the IP addresses on the SR creation request and match them with the equivalent PIFs.

For example here is the output of the xe network-param-list to retrieve the associated PIF's and the xe pif-param-lst with the UUID's of the retrieved PIFs:

[***@xenserver1 ~]# xe network-param-list uuid=4dc936d7-e806-c69a-a292-78e0ed2d5faa
uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa
name-label ( RW): iSCSI #0
name-description ( RW):
VIF-uuids (SRO):
PIF-uuids (SRO): 3baa8436-feca-cd04-3173-5002d9ffc66b; 362d63cb-647a-465f-6b81-1b59cd3b1f5d
MTU ( RW): 9000
bridge ( RO): xenbr2
other-config (MRW): automatic: true
blobs ( RO):
tags (SRW):
default-locking-mode ( RW): unlocked


[***@xenserver1 ~]# xe pif-param-list uuid=3baa8436-feca-cd04-3173-5002d9ffc66b
uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5c:64
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-master-of ( RO):
bond-slave-of ( RO): <not in database>
tunnel-access-PIF-of ( RO):
tunnel-transport-PIF-of ( RO):
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
host-name-label ( RO): xenserver2.local.iq.ufrj.br
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.20.2
netmask ( RO): 255.255.255.252
gateway ( RO):
IPv6-configuration-mode ( RO): None
IPv6 ( RO):
IPv6-gateway ( RO):
primary-address-type ( RO): IPv4
DNS ( RO):
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0


[***@xenserver1 ~]# xe pif-param-list uuid=362d63cb-647a-465f-6b81-1b59cd3b1f5d
uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5a:1c
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-master-of ( RO):
bond-slave-of ( RO): <not in database>
tunnel-access-PIF-of ( RO):
tunnel-transport-PIF-of ( RO):
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c
host-name-label ( RO): xenserver1.local.iq.ufrj.br
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.10.2
netmask ( RO): 255.255.255.252
gateway ( RO):
IPv6-configuration-mode ( RO): None
IPv6 ( RO):
IPv6-gateway ( RO):
primary-address-type ( RO): IPv4
DNS ( RO):
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0

So as you can see, we have the IP addresses and the network mask. Which is sufficient to an efficient and precise algorithm of SR creation.

I thinks this clarifies everything we discussed until now. A patch on XenCenter appears to be really viable.

Many thanks,
Vinicius.
Yes, correct. The MD36xx looks like two separate controller units, each acting as a separate device. For a standalone server or some storage devices, that's right that you need a separate subnet for each connection.
Because a single connection isn't seeing the broadcast from the other connectors at the time of the initial connection from one host, the pool is unaware of the others. The "repair" operation causes this to be conducted from each host, and as I guessed, then fixes the SR connectivity for the entire pool. I am speculating that the request for a new connection of this type via XenCenter could perhaps be solved by propagating that initial connection process to the rest of the hosts in the pool and then rescan the SR.
-=Tobias
Post by Vinícius Ferrão
Hi Tobias,
As for I know, the Dell MD36xx is a dual controller based storage. So it's basically "two computers". Considering this with the added knowledge that I've about TCP networking there's a golden rule that says: only one IP address on a given subnet on a single computer. So I can't use the same network on my case. That's why I use point-to-point links too.
Since I've only one machine acting as a storage server (It's a Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do need four separate networks. FreeNAS is FreeBSD based, and BSD enforces this rule by default. I can't have two IP addresses of the same subnet on same machine, even if I manually configure it.
If this wasn't bad TCP network practice, I would be able to configure the SR via XenCenter, since it will "find" the same network on the others hosts without problem.
About the XenCenter, it really appears to be only a missing function on the software. If I was a good programmer I would try to implement this. But this is beyond my knowledge... :(
Thanks once again,
Vinícius.
No reason to necessarily use four separate subnets. You could have 192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and 2.2 on the other controller (on, for example, an MD36xx that is standard). If however it is something like an MD32xx which recommends four separate subets, then of course, use four. One should follow the vendor's specifications.
I will respond in more detail later when I get a chance, Vinícius, but after a quick read, I agree with your findings from your other email. It would appear that the initial connection is indeed evidently a missing function in XenCenter and perhaps all that is needed to make this work "out of the box". As to the wildcard discovery, it's generally preferred as it allows the host to figure out the path on its own. The only reason I could see not using a wildcard discovery would be if there were any chance for overlaps and ambiguous connections.
Regards,
--Tobias
Post by Vinícius Ferrão
Hello Dave,
+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+
192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30
192.168.10.2/30
192.168.11.2/30
192.168.20.2/30
192.168.21.2/30
That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0
Cheers,
Vinícius.
Hi,
Post by Vinícius Ferrão
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked
 Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0
I’m really excited because it worked. So let’s me explain what I’ve did.
First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I’d like to understand your configuration better— have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?
Thanks,
Dave
Post by Vinícius Ferrão
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0
I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Post by Vinícius Ferrão
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/
It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Tobias Kreidl
2015-02-17 00:55:07 UTC
Permalink
Vinícius,
The primary difference is with an SR associated with a storage device
that is connected via a switch is that the connection to that storage
device will reveal any other IP addresses associated with the primary
interface on that storage device. So, it basically "works" because the
XenServers see all the addresses being broadcast back. With the initial
request coming from a single host and the information not being returned
for the other IP addresses to the other pool members (because of no
general broadcast), there is therefore no mechanism that triggers the
discovery on the other XenServers within the pool.

So, as mentioned before, one option would be to look if multiple
addresses are somehow specified on the initial connection and use that
to trigger pushing the connection request to go to the other hosts in
the pool. This is in essence what the sr-repair operation accomplishes,
namely it looks to see if there is a PBD associated with the SR on each
host within the pool. If so, it tries to plug it in. If not, it will
issue a pbd-create command pointing to the specific SR and then try to
plug in the PBD. So, in essence, you'd want to trigger that SR repair
sequence automatically given this situation. That would still evidently
require a modification to the SR creation wizard.

-=Tobias
Post by Vinícius Ferrão
Tobias,
I've done more tests about the SR creation over XenCenter.
All the process definitely occurs only on the pool master. Even if the
specified networks are not part of the networks of the host. The
connection is tried on the default route.
192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1
The target IQN is discovered without any problem, since the iSCSI
Storage reports the LUNs on any interface.
But when trying to discover the LUN, the process only is done on the
pool master. During the discovery process, I left a watch command on
tcp 0 0 10.7.0.101:443 192.168.172.1:58053
ESTABLISHED
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365
ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058
ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266
ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930
ESTABLISHED
tcp 0 48 10.7.0.101:22 192.168.172.1:58064
ESTABLISHED
tcp 0 0 192.168.10.2:55870 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:52654 192.168.11.1:3260
TIME_WAIT
tcp 0 0 10.7.0.101:443 192.168.172.1:58055
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054
ESTABLISHED
tcp 0 0 192.168.11.2:52656 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260
TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
During the discovery, look at the 192.168.21.1 address being searched
tcp 0 0 10.7.0.101:443 192.168.172.1:58053
ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365
ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058
ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266
ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930
ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260
TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064
ESTABLISHED
******************
tcp 0 1 10.7.0.101:56481 192.168.21.1:3260
SYN_SENT
******************
tcp 0 0 10.7.0.101:443 192.168.172.1:58055
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054
ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260
TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
Even addresses that does not exist on the pool (since the IQN
tcp 0 0 10.7.0.101:443 192.168.172.1:58053
ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365
ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058
ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266
ESTABLISHED
******************
tcp 0 1 10.7.0.101:52787 192.168.30.1:3260
SYN_SENT
******************
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930
ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260
TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58055
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054
ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260
TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
If you're wondering the 192.168.30.1 is reported because in the future
we will add another XS host to the pool. The point here is that
XenCenter should not lookup for this network since it isn't specified
during the Software iSCSI creation. On my understanding it should only
lookup for the specified addresses.
And to finish the tests, the second XS host remains equally during all
tcp 0 48 10.7.0.102:22 192.168.172.1:58046
ESTABLISHED
udp 0 0 192.168.21.2:123 0.0.0.0:*
udp 0 0 192.168.20.2:123 0.0.0.0:*
As you said, perhaps propagating the initial connection through the
host will solve the problem without modifying the XenCenter interface.
A more sophisticated approach would be query the network-list over
XAPI, retrieve the PIFs associated to those networks and them just
check the IP addresses on the SR creation request and match them with
the equivalent PIFs.
For example here is the output of the xe network-param-list to
retrieve the associated PIF's and the xe pif-param-lst with the UUID's
uuid=4dc936d7-e806-c69a-a292-78e0ed2d5faa
uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa
name-label ( RW): iSCSI #0
PIF-uuids (SRO): 3baa8436-feca-cd04-3173-5002d9ffc66b;
362d63cb-647a-465f-6b81-1b59cd3b1f5d
MTU ( RW): 9000
bridge ( RO): xenbr2
other-config (MRW): automatic: true
default-locking-mode ( RW): unlocked
uuid=3baa8436-feca-cd04-3173-5002d9ffc66b
uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5c:64
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
host-name-label ( RO): xenserver2.local.iq.ufrj.br
<http://xenserver2.local.iq.ufrj.br>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.20.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
uuid=362d63cb-647a-465f-6b81-1b59cd3b1f5d
uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5a:1c
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c
host-name-label ( RO): xenserver1.local.iq.ufrj.br
<http://xenserver1.local.iq.ufrj.br>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.10.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
So as you can see, we have the IP addresses and the network mask.
Which is sufficient to an efficient and precise algorithm of SR creation.
I thinks this clarifies everything we discussed until now. A patch on
XenCenter appears to be really viable.
Many thanks,
Vinicius.
Post by Tobias Kreidl
Yes, correct. The MD36xx looks like two separate controller units,
each acting as a separate device. For a standalone server or some
storage devices, that's right that you need a separate subnet for
each connection.
Because a single connection isn't seeing the broadcast from the other
connectors at the time of the initial connection from one host, the
pool is unaware of the others. The "repair" operation causes this to
be conducted from each host, and as I guessed, then fixes the SR
connectivity for the entire pool. I am speculating that the request
for a new connection of this type via XenCenter could perhaps be
solved by propagating that initial connection process to the rest of
the hosts in the pool and then rescan the SR.
-=Tobias
Post by Vinícius Ferrão
Hi Tobias,
As for I know, the Dell MD36xx is a dual controller based storage.
So it's basically "two computers". Considering this with the added
knowledge that I've about TCP networking there's a golden rule that
says: only one IP address on a given subnet on a single computer. So
I can't use the same network on my case. That's why I use
point-to-point links too.
Since I've only one machine acting as a storage server (It's a
Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do
need four separate networks. FreeNAS is FreeBSD based, and BSD
enforces this rule by default. I can't have two IP addresses of the
same subnet on same machine, even if I manually configure it.
If this wasn't bad TCP network practice, I would be able to
configure the SR via XenCenter, since it will "find" the same
network on the others hosts without problem.
About the XenCenter, it really appears to be only a missing function
on the software. If I was a good programmer I would try to implement
this. But this is beyond my knowledge... :(
Thanks once again,
Vinícius.
Post by Tobias Kreidl
No reason to necessarily use four separate subnets. You could have
192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and
2.2 on the other controller (on, for example, an MD36xx that is
standard). If however it is something like an MD32xx which
recommends four separate subets, then of course, use four. One
should follow the vendor's specifications.
I will respond in more detail later when I get a chance, Vinícius,
but after a quick read, I agree with your findings from your other
email. It would appear that the initial connection is indeed
evidently a missing function in XenCenter and perhaps all that is
needed to make this work "out of the box". As to the wildcard
discovery, it's generally preferred as it allows the host to figure
out the path on its own. The only reason I could see not using a
wildcard discovery would be if there were any chance for overlaps
and ambiguous connections.
Regards,
--Tobias
Post by Vinícius Ferrão
Hello Dave,
In total I have four paths. Two in each host. I've made a drawning
+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+
192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30
192.168.10.2/30
192.168.11.2/30
192.168.20.2/30
192.168.21.2/30
That's why I'm using multipath. Because I do need MPIO to sum the
two network interfaces per host.
To exemplify a little more heres a screenshot of the network
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0
Cheers,
Vinícius.
Hi,
On 15 Feb 2015, at 05:44, Vinícius Ferrão
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do
it, they recommend, and they even sell a solution using DELL
hardware with two machines and a shared storage running with
10GbE links. Point-ot-point networks, dual controllers on the
storage side and two different network for iSCSI. They sell it
at least here in Brazil. What I’m trying to do is achieve the
same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a
better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different
markets. For example, I’ve used Fibre Channel only one time in
my life. And, believe it or not, it was a switchless
configuration. And it worked
 Wasn’t for VM workload but was in
a GRID Cluster with TORQUE/OpenPBS software. I do have the
pictures of the topology, but I need to search for it. At that
time, I wasn’t the guy who created the solution, but I
understood that it was a huge money saving, because the FC
switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk
technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was
necessary in the CLI, but I was comfortable with this. The
https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0
I’m really excited because it worked. So let’s me explain what
I’ve did.
First I started creating the SR via XenCenter, it failed as
expected. But to bypass I just created the SR via XenCenter with
the two Storage IP’s of the Pool Master. So the process went OK,
but it started broken, since the second XS host cannot see the
IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the
SR UUID created by XenCenter and the referenced PBDs. According
to what I know about XS, the PBD’s are the physical connect from
separate XS hosts to the shared storage. So it’s not related to
other hosts, it’s exclusively to the host machine that I’m
using. Understood that, the ideia is to destroy the created PBD
on the second XS host and recreate it with the proper settings
and IP addresses.
That’s when things got interesting. To create the PBD, I do need
the iSCSI working in the host, and consequently the multipath
connection. So I done the process in the terminal with the
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p
192.168.20.1:3260 --login
iscsiadm -m node --targetname
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p
192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
After all I was able to create the PBD with those new settings.
The tricky part was to pass the required “device-config” values
via the xe pbd-create command. I wasn’t aware of the method, but
after 4 tries I understood the process and typed the following
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f
host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|"
device-config:target=192.168.20.1,192.168.21.1
device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260
device-config:targetIQN=*
device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000
device-config:port=3260
I’d like to understand your configuration better— have you
effectively enabled 2 storage paths for the SR, knowing that each
host will only be able to use one path? Are your PBDs identical
or are there differences?
Thanks,
Dave
Basically everything necessary is in there. The IP’s, the name,
the multipath link, the LUN values, everything. After this, I
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
To prove that’s everything is working fine, I’ve created a new
VM, installed FreeBSD 10 on it, done a pkg install
xe-guest-utilities and requested a XenMotion operation. Oh boy,
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0
I thinks this is sufficient for now, to prove that the concept
works and XenServer can create this kind of topology. It’s just
not available on the GUI. If XenCenter supports this kind of SR
on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was
really anxious to report this in the list.
On Feb 14, 2015, at 8:14 PM, Tobias Kreidl
Frankly speaking, it doesn't seem that it's a good idea to try
to try to squeeze something out of a technology that is not
built-in or future-safe. Would you want to try to connect a
fiber-channel storage device without a fiber channel switch (a
very similar situation)? There are other alternatives, for
example, to connect iSCSi storage to another,
XenServer-independent server and then serve storage from it via
NFS or use NFS in the first place and not make use at all of
iSCSI technology. NFS has come a long way and in many cases,
can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool
are a significant consideration and cannot be ignored as part
of the storage support protocols within XenServer. XenServer
provides a number of options for storage connectivity to cover
a wide range of needs, preferences and costs. Often, trying to
save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism
altogether (obviously, not a supported configuration) and
connect at least Linux VMs directly to iSCSI storage using open
iSCSI, but I have not explored this without still involving a
network switch. As you indicated, it's also not clear if
XenServer will remain 100% open iSCSI compatible or not in the
future. It also takes aways some of the things an SR can offer,
like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors
in in improving performance. Networks switches are much more
capable and much faster than any storage device or host in
routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Post by Vinícius Ferrão
Hello guys,
I was talking with Tim Mackey on Twitter about the viability
of using a iSCSI SR without a Switch. The main goal is to
reduce the TCO of the solution, get some extra performance
removing the overhead that a managed switch puts on the
network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those
kind of setup with a distributed switch, so DVSC should be an
option. But as Tim said it’s only a interface to control some
policies.
I’ve looked for help on ServerFault and FreeNAS Forums, since
I’m running FreeNAS as iSCSI service. The discussion is going
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/
It appears that XenServer cannot support switchless iSCSI
storages without a switch, because XS needs the same network
block on all the XS hosts to create a shared storage between
the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the
appropriate PBD’s. Perhaps one way to solve this issue is
ignoring XenCenter on SR creation and try to create an SR with
all the PBD’s from the XS hosts. But I was unable to test this
setting due to lack of my knowledge when trying to create and
manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback
of the developers, to see if this method of building the iSCSI
network is viable, since as Tim said, XS is not 100%
open-iscsi upstream and if it could be integrated on next
versions of XenServer via the XenCenter GUI. I think that’s a
very common architecture and VMware is doing this on small
pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Vinícius Ferrão
2015-02-17 20:09:21 UTC
Permalink
Hello again Tobias,

I've tried to change the code by myself, but unfortunately I was unable to achieve this. I was able to git clone the source, find the deps, compile and run XenCenter in debug mode, but the high level of the programming language is at this moment ahead of my knowledge. I'm not a real world programmer, just an tcsh and ANSI C guy.

I've managed to trace the problem up to the XenAPI.SR.async_create(); function. But reading the XenAPI code was a no-go for me, mainly when I reached this:

public static XenRef<Task> async_create(Session session, string _host, Dictionary<string, string> _device_config, long _physical_size, string _name_label, string _name_description, string _type, string _content_type, bool _shared, Dictionary<string, string> _sm_config)
{
return XenRef<Task>.Create(session.proxy.async_sr_create(session.uuid, (_host != null) ? _host : "", Maps.convert_to_proxy_string_string(_device_config), _physical_size.ToString(), (_name_label != null) ? _name_label : "", (_name_description != null) ? _name_description : "", (_type != null) ? _type : "", (_content_type != null) ? _content_type : "", _shared, Maps.convert_to_proxy_string_string(_sm_config)).parse());
}

The point here is: there's any place I can request for an enhancement on XenCenter? It's possible for guys without the Citrix paid support?

Thanks in advance,
Vinicius.
Vinícius,
The primary difference is with an SR associated with a storage device that is connected via a switch is that the connection to that storage device will reveal any other IP addresses associated with the primary interface on that storage device. So, it basically "works" because the XenServers see all the addresses being broadcast back. With the initial request coming from a single host and the information not being returned for the other IP addresses to the other pool members (because of no general broadcast), there is therefore no mechanism that triggers the discovery on the other XenServers within the pool.
So, as mentioned before, one option would be to look if multiple addresses are somehow specified on the initial connection and use that to trigger pushing the connection request to go to the other hosts in the pool. This is in essence what the sr-repair operation accomplishes, namely it looks to see if there is a PBD associated with the SR on each host within the pool. If so, it tries to plug it in. If not, it will issue a pbd-create command pointing to the specific SR and then try to plug in the PBD. So, in essence, you'd want to trigger that SR repair sequence automatically given this situation. That would still evidently require a modification to the SR creation wizard.
-=Tobias
Post by Vinícius Ferrão
Tobias,
I've done more tests about the SR creation over XenCenter.
All the process definitely occurs only on the pool master. Even if the specified networks are not part of the networks of the host. The connection is tried on the default route.
192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1
The target IQN is discovered without any problem, since the iSCSI Storage reports the LUNs on any interface.
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 192.168.10.2:55870 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52654 192.168.11.1:3260 TIME_WAIT
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.11.2:52656 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
******************
tcp 0 1 10.7.0.101:56481 192.168.21.1:3260 SYN_SENT
******************
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
******************
tcp 0 1 10.7.0.101:52787 192.168.30.1:3260 SYN_SENT
******************
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
If you're wondering the 192.168.30.1 is reported because in the future we will add another XS host to the pool. The point here is that XenCenter should not lookup for this network since it isn't specified during the Software iSCSI creation. On my understanding it should only lookup for the specified addresses.
tcp 0 48 10.7.0.102:22 192.168.172.1:58046 ESTABLISHED
udp 0 0 192.168.21.2:123 0.0.0.0:*
udp 0 0 192.168.20.2:123 0.0.0.0:*
As you said, perhaps propagating the initial connection through the host will solve the problem without modifying the XenCenter interface. A more sophisticated approach would be query the network-list over XAPI, retrieve the PIFs associated to those networks and them just check the IP addresses on the SR creation request and match them with the equivalent PIFs.
uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa
name-label ( RW): iSCSI #0
PIF-uuids (SRO): 3baa8436-feca-cd04-3173-5002d9ffc66b; 362d63cb-647a-465f-6b81-1b59cd3b1f5d
MTU ( RW): 9000
bridge ( RO): xenbr2
other-config (MRW): automatic: true
default-locking-mode ( RW): unlocked
uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5c:64
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
host-name-label ( RO): xenserver2.local.iq.ufrj.br <http://xenserver2.local.iq.ufrj.br/>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.20.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5a:1c
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c
host-name-label ( RO): xenserver1.local.iq.ufrj.br <http://xenserver1.local.iq.ufrj.br/>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.10.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
So as you can see, we have the IP addresses and the network mask. Which is sufficient to an efficient and precise algorithm of SR creation.
I thinks this clarifies everything we discussed until now. A patch on XenCenter appears to be really viable.
Many thanks,
Vinicius.
Yes, correct. The MD36xx looks like two separate controller units, each acting as a separate device. For a standalone server or some storage devices, that's right that you need a separate subnet for each connection.
Because a single connection isn't seeing the broadcast from the other connectors at the time of the initial connection from one host, the pool is unaware of the others. The "repair" operation causes this to be conducted from each host, and as I guessed, then fixes the SR connectivity for the entire pool. I am speculating that the request for a new connection of this type via XenCenter could perhaps be solved by propagating that initial connection process to the rest of the hosts in the pool and then rescan the SR.
-=Tobias
Post by Vinícius Ferrão
Hi Tobias,
As for I know, the Dell MD36xx is a dual controller based storage. So it's basically "two computers". Considering this with the added knowledge that I've about TCP networking there's a golden rule that says: only one IP address on a given subnet on a single computer. So I can't use the same network on my case. That's why I use point-to-point links too.
Since I've only one machine acting as a storage server (It's a Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do need four separate networks. FreeNAS is FreeBSD based, and BSD enforces this rule by default. I can't have two IP addresses of the same subnet on same machine, even if I manually configure it.
If this wasn't bad TCP network practice, I would be able to configure the SR via XenCenter, since it will "find" the same network on the others hosts without problem.
About the XenCenter, it really appears to be only a missing function on the software. If I was a good programmer I would try to implement this. But this is beyond my knowledge... :(
Thanks once again,
Vinícius.
No reason to necessarily use four separate subnets. You could have 192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and 2.2 on the other controller (on, for example, an MD36xx that is standard). If however it is something like an MD32xx which recommends four separate subets, then of course, use four. One should follow the vendor's specifications.
I will respond in more detail later when I get a chance, Vinícius, but after a quick read, I agree with your findings from your other email. It would appear that the initial connection is indeed evidently a missing function in XenCenter and perhaps all that is needed to make this work "out of the box". As to the wildcard discovery, it's generally preferred as it allows the host to figure out the path on its own. The only reason I could see not using a wildcard discovery would be if there were any chance for overlaps and ambiguous connections.
Regards,
--Tobias
Post by Vinícius Ferrão
Hello Dave,
+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+
192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30
192.168.10.2/30
192.168.11.2/30
192.168.20.2/30
192.168.21.2/30
That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0 <Loading Image...
Cheers,
Vinícius.
Hi,
Post by Vinícius Ferrão
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked
 Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0 <https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0>
I’m really excited because it worked. So let’s me explain what I’ve did.
First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I’d like to understand your configuration better— have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?
Thanks,
Dave
Post by Vinícius Ferrão
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0 <https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0>
I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Post by Vinícius Ferrão
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch <http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch>
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/ <https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/>
It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Tobias Kreidl
2015-02-17 20:16:01 UTC
Permalink
Here is one place you can request a feature/enhancement:
http://discussions.citrix.com/forum/1289-feature-requests-and-suggestions/

Unfortunately, I do not have time to take a closer look at it myself at
this level, as interesting as it is.

-=Tobias
Post by Vinícius Ferrão
Hello again Tobias,
I've tried to change the code by myself, but unfortunately I was
unable to achieve this. I was able to git clone the source, find the
deps, compile and run XenCenter in debug mode, but the high level of
the programming language is at this moment ahead of my knowledge. I'm
not a real world programmer, just an tcsh and ANSI C guy.
I've managed to trace the problem up to the XenAPI.SR.async_create();
function. But reading the XenAPI code was a no-go for me, mainly when
public static XenRef<Task> async_create(Session session, string _host,
Dictionary<string, string> _device_config, long _physical_size, string
_name_label, string _name_description, string _type,
string _content_type, bool _shared, Dictionary<string, string> _sm_config)
{
return
XenRef<Task>.Create(session.proxy.async_sr_create(session.uuid, (_host
"", Maps.convert_to_proxy_string_string(_device_config),
_physical_size.ToString(), (_name_label != null) ? _name_label : "",
(_name_description != null) ? _name_description : "", (_type != null)
? _type : "", (_content_type != null) ? _content_type : "",
_shared, Maps.convert_to_proxy_string_string(_sm_config)).parse());
}
The point here is: there's any place I can request for an enhancement
on XenCenter? It's possible for guys without the Citrix paid support?
Thanks in advance,
Vinicius.
Vinícius,
The primary difference is with an SR associated with a storage device
that is connected via a switch is that the connection to that storage
device will reveal any other IP addresses associated with the primary
interface on that storage device. So, it basically "works" because
the XenServers see all the addresses being broadcast back. With the
initial request coming from a single host and the information not
being returned for the other IP addresses to the other pool members
(because of no general broadcast), there is therefore no mechanism
that triggers the discovery on the other XenServers within the pool.
So, as mentioned before, one option would be to look if multiple
addresses are somehow specified on the initial connection and use
that to trigger pushing the connection request to go to the other
hosts in the pool. This is in essence what the sr-repair operation
accomplishes, namely it looks to see if there is a PBD associated
with the SR on each host within the pool. If so, it tries to plug it
in. If not, it will issue a pbd-create command pointing to the
specific SR and then try to plug in the PBD. So, in essence, you'd
want to trigger that SR repair sequence automatically given this
situation. That would still evidently require a modification to the
SR creation wizard.
-=Tobias
Post by Vinícius Ferrão
Tobias,
I've done more tests about the SR creation over XenCenter.
All the process definitely occurs only on the pool master. Even if
the specified networks are not part of the networks of the host. The
connection is tried on the default route.
192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1
The target IQN is discovered without any problem, since the iSCSI
Storage reports the LUNs on any interface.
But when trying to discover the LUN, the process only is done on the
pool master. During the discovery process, I left a watch command on
tcp 0 0 10.7.0.101:443 192.168.172.1:58053
ESTABLISHED
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365
ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058
ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266
ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930
ESTABLISHED
tcp 0 48 10.7.0.101:22 192.168.172.1:58064
ESTABLISHED
tcp 0 0 192.168.10.2:55870 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:52654 192.168.11.1:3260
TIME_WAIT
tcp 0 0 10.7.0.101:443 192.168.172.1:58055
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054
ESTABLISHED
tcp 0 0 192.168.11.2:52656 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260
TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
During the discovery, look at the 192.168.21.1 address being
tcp 0 0 10.7.0.101:443 192.168.172.1:58053
ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365
ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058
ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266
ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930
ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260
TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064
ESTABLISHED
******************
tcp 0 1 10.7.0.101:56481 192.168.21.1:3260
SYN_SENT
******************
tcp 0 0 10.7.0.101:443 192.168.172.1:58055
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054
ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260
TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
Even addresses that does not exist on the pool (since the IQN
tcp 0 0 10.7.0.101:443 192.168.172.1:58053
ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365
ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058
ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266
ESTABLISHED
******************
tcp 0 1 10.7.0.101:52787 192.168.30.1:3260
SYN_SENT
******************
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930
ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260
TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58055
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054
ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260
TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
If you're wondering the 192.168.30.1 is reported because in the
future we will add another XS host to the pool. The point here is
that XenCenter should not lookup for this network since it isn't
specified during the Software iSCSI creation. On my understanding it
should only lookup for the specified addresses.
And to finish the tests, the second XS host remains equally during
tcp 0 48 10.7.0.102:22 192.168.172.1:58046
ESTABLISHED
udp 0 0 192.168.21.2:123 0.0.0.0:*
udp 0 0 192.168.20.2:123 0.0.0.0:*
As you said, perhaps propagating the initial connection through the
host will solve the problem without modifying the XenCenter
interface. A more sophisticated approach would be query the
network-list over XAPI, retrieve the PIFs associated to those
networks and them just check the IP addresses on the SR creation
request and match them with the equivalent PIFs.
For example here is the output of the xe network-param-list to
retrieve the associated PIF's and the xe pif-param-lst with the
uuid=4dc936d7-e806-c69a-a292-78e0ed2d5faa
uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa
name-label ( RW): iSCSI #0
3baa8436-feca-cd04-3173-5002d9ffc66b;
362d63cb-647a-465f-6b81-1b59cd3b1f5d
MTU ( RW): 9000
bridge ( RO): xenbr2
other-config (MRW): automatic: true
default-locking-mode ( RW): unlocked
uuid=3baa8436-feca-cd04-3173-5002d9ffc66b
uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5c:64
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
host-name-label ( RO): xenserver2.local.iq.ufrj.br
<http://xenserver2.local.iq.ufrj.br/>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.20.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
uuid=362d63cb-647a-465f-6b81-1b59cd3b1f5d
uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5a:1c
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c
host-name-label ( RO): xenserver1.local.iq.ufrj.br
<http://xenserver1.local.iq.ufrj.br/>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.10.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
So as you can see, we have the IP addresses and the network mask.
Which is sufficient to an efficient and precise algorithm of SR creation.
I thinks this clarifies everything we discussed until now. A patch
on XenCenter appears to be really viable.
Many thanks,
Vinicius.
Post by Tobias Kreidl
Yes, correct. The MD36xx looks like two separate controller units,
each acting as a separate device. For a standalone server or some
storage devices, that's right that you need a separate subnet for
each connection.
Because a single connection isn't seeing the broadcast from the
other connectors at the time of the initial connection from one
host, the pool is unaware of the others. The "repair" operation
causes this to be conducted from each host, and as I guessed, then
fixes the SR connectivity for the entire pool. I am speculating
that the request for a new connection of this type via XenCenter
could perhaps be solved by propagating that initial connection
process to the rest of the hosts in the pool and then rescan the SR.
-=Tobias
Post by Vinícius Ferrão
Hi Tobias,
As for I know, the Dell MD36xx is a dual controller based storage.
So it's basically "two computers". Considering this with the added
knowledge that I've about TCP networking there's a golden rule
that says: only one IP address on a given subnet on a single
computer. So I can't use the same network on my case. That's why I
use point-to-point links too.
Since I've only one machine acting as a storage server (It's a
Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do
need four separate networks. FreeNAS is FreeBSD based, and BSD
enforces this rule by default. I can't have two IP addresses of
the same subnet on same machine, even if I manually configure it.
If this wasn't bad TCP network practice, I would be able to
configure the SR via XenCenter, since it will "find" the same
network on the others hosts without problem.
About the XenCenter, it really appears to be only a missing
function on the software. If I was a good programmer I would try
to implement this. But this is beyond my knowledge... :(
Thanks once again,
Vinícius.
Post by Tobias Kreidl
No reason to necessarily use four separate subnets. You could
have 192.168.10.1. and 20.1 on one iSCSI controller and
192.168.10.2 and 2.2 on the other controller (on, for example, an
MD36xx that is standard). If however it is something like an
MD32xx which recommends four separate subets, then of course, use
four. One should follow the vendor's specifications.
I will respond in more detail later when I get a chance,
Vinícius, but after a quick read, I agree with your findings from
your other email. It would appear that the initial connection is
indeed evidently a missing function in XenCenter and perhaps all
that is needed to make this work "out of the box". As to the
wildcard discovery, it's generally preferred as it allows the
host to figure out the path on its own. The only reason I could
see not using a wildcard discovery would be if there were any
chance for overlaps and ambiguous connections.
Regards,
--Tobias
Post by Vinícius Ferrão
Hello Dave,
In total I have four paths. Two in each host. I've made a
drawning using ASCII characters, hope this can clarify the
+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+
The Storage has four gigabit ethernet adapters with the
192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30
192.168.10.2/30
192.168.11.2/30
192.168.20.2/30
192.168.21.2/30
That's why I'm using multipath. Because I do need MPIO to sum
the two network interfaces per host.
To exemplify a little more heres a screenshot of the network
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0
Cheers,
Vinícius.
On Feb 15, 2015, at 12:22 PM, Dave Scott
Hi,
On 15 Feb 2015, at 05:44, Vinícius Ferrão
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do
it, they recommend, and they even sell a solution using DELL
hardware with two machines and a shared storage running with
10GbE links. Point-ot-point networks, dual controllers on the
storage side and two different network for iSCSI. They sell it
at least here in Brazil. What I’m trying to do is achieve the
same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is
a better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different
markets. For example, I’ve used Fibre Channel only one time in
my life. And, believe it or not, it was a switchless
configuration. And it worked
 Wasn’t for VM workload but was
in a GRID Cluster with TORQUE/OpenPBS software. I do have the
pictures of the topology, but I need to search for it. At that
time, I wasn’t the guy who created the solution, but I
understood that it was a huge money saving, because the FC
switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk
technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was
necessary in the CLI, but I was comfortable with this. The
https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0
I’m really excited because it worked. So let’s me explain what
I’ve did.
First I started creating the SR via XenCenter, it failed as
expected. But to bypass I just created the SR via XenCenter
with the two Storage IP’s of the Pool Master. So the process
went OK, but it started broken, since the second XS host
cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the
SR UUID created by XenCenter and the referenced PBDs.
According to what I know about XS, the PBD’s are the physical
connect from separate XS hosts to the shared storage. So it’s
not related to other hosts, it’s exclusively to the host
machine that I’m using. Understood that, the ideia is to
destroy the created PBD on the second XS host and recreate it
with the proper settings and IP addresses.
That’s when things got interesting. To create the PBD, I do
need the iSCSI working in the host, and consequently the
multipath connection. So I done the process in the terminal
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p
192.168.20.1:3260 --login
iscsiadm -m node --targetname
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p
192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
After all I was able to create the PBD with those new
settings. The tricky part was to pass the required
“device-config” values via the xe pbd-create command. I wasn’t
aware of the method, but after 4 tries I understood the
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f
host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|"
device-config:target=192.168.20.1,192.168.21.1
device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260
device-config:targetIQN=*
device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000
device-config:port=3260
I’d like to understand your configuration better— have you
effectively enabled 2 storage paths for the SR, knowing that
each host will only be able to use one path? Are your PBDs
identical or are there differences?
Thanks,
Dave
Basically everything necessary is in there. The IP’s, the
name, the multipath link, the LUN values, everything. After
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
To prove that’s everything is working fine, I’ve created a new
VM, installed FreeBSD 10 on it, done a pkg install
xe-guest-utilities and requested a XenMotion operation. Oh
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0
I thinks this is sufficient for now, to prove that the concept
works and XenServer can create this kind of topology. It’s
just not available on the GUI. If XenCenter supports this kind
of SR on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I
was really anxious to report this in the list.
On Feb 14, 2015, at 8:14 PM, Tobias Kreidl
Frankly speaking, it doesn't seem that it's a good idea to
try to try to squeeze something out of a technology that is
not built-in or future-safe. Would you want to try to connect
a fiber-channel storage device without a fiber channel switch
(a very similar situation)? There are other alternatives,
for example, to connect iSCSi storage to another,
XenServer-independent server and then serve storage from it
via NFS or use NFS in the first place and not make use at all
of iSCSI technology. NFS has come a long way and in many
cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a
pool are a significant consideration and cannot be ignored as
part of the storage support protocols within XenServer.
XenServer provides a number of options for storage
connectivity to cover a wide range of needs, preferences and
costs. Often, trying to save money in the wrong area results
in more headaches later on.
That all being said, it is possible to bypass the SR
mechanism altogether (obviously, not a supported
configuration) and connect at least Linux VMs directly to
iSCSI storage using open iSCSI, but I have not explored this
without still involving a network switch. As you indicated,
it's also not clear if XenServer will remain 100% open iSCSI
compatible or not in the future. It also takes aways some of
the things an SR can offer, like VM cloning, storage
migration, etc.
Finally, I am not sure I see where the lack of a switch
factors in in improving performance. Networks switches are
much more capable and much faster than any storage device or
host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Post by Vinícius Ferrão
Hello guys,
I was talking with Tim Mackey on Twitter about the viability
of using a iSCSI SR without a Switch. The main goal is to
reduce the TCO of the solution, get some extra performance
removing the overhead that a managed switch puts on the
network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those
kind of setup with a distributed switch, so DVSC should be
an option. But as Tim said it’s only a interface to control
some policies.
I’ve looked for help on ServerFault and FreeNAS Forums,
since I’m running FreeNAS as iSCSI service. The discussion
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/
It appears that XenServer cannot support switchless iSCSI
storages without a switch, because XS needs the same network
block on all the XS hosts to create a shared storage between
the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the
appropriate PBD’s. Perhaps one way to solve this issue is
ignoring XenCenter on SR creation and try to create an SR
with all the PBD’s from the XS hosts. But I was unable to
test this setting due to lack of my knowledge when trying to
create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback
of the developers, to see if this method of building the
iSCSI network is viable, since as Tim said, XS is not 100%
open-iscsi upstream and if it could be integrated on next
versions of XenServer via the XenCenter GUI. I think that’s
a very common architecture and VMware is doing this on small
pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Stephen Turner
2015-02-18 10:15:29 UTC
Permalink
I admit I haven’t fully understood the whole scenario here, but I’m not sure that XenCenter would be where it would have to be changed. It seems to me that it would probably be in the SR creation code on the server side.
--
Stephen Turner


From: xs-devel-***@lists.xenserver.org [mailto:xs-devel-***@lists.xenserver.org] On Behalf Of Vinícius Ferrão
Sent: 17 February 2015 20:09
To: Tobias Kreidl
Cc: xs-***@lists.xenserver.org
Subject: Re: [xs-devel] Switchless shared iSCSI Network with /30 links

Hello again Tobias,

I've tried to change the code by myself, but unfortunately I was unable to achieve this. I was able to git clone the source, find the deps, compile and run XenCenter in debug mode, but the high level of the programming language is at this moment ahead of my knowledge. I'm not a real world programmer, just an tcsh and ANSI C guy.

I've managed to trace the problem up to the XenAPI.SR.async_create(); function. But reading the XenAPI code was a no-go for me, mainly when I reached this:

public static XenRef<Task> async_create(Session session, string _host, Dictionary<string, string> _device_config, long _physical_size, string _name_label, string _name_description, string _type, string _content_type, bool _shared, Dictionary<string, string> _sm_config)
{
return XenRef<Task>.Create(session.proxy.async_sr_create(session.uuid, (_host != null) ? _host : "", Maps.convert_to_proxy_string_string(_device_config), _physical_size.ToString(), (_name_label != null) ? _name_label : "", (_name_description != null) ? _name_description : "", (_type != null) ? _type : "", (_content_type != null) ? _content_type : "", _shared, Maps.convert_to_proxy_string_string(_sm_config)).parse());
}

The point here is: there's any place I can request for an enhancement on XenCenter? It's possible for guys without the Citrix paid support?

Thanks in advance,
Vinicius.

On Feb 16, 2015, at 10:55 PM, Tobias Kreidl <***@nau.edu<mailto:***@nau.edu>> wrote:

Vinícius,
The primary difference is with an SR associated with a storage device that is connected via a switch is that the connection to that storage device will reveal any other IP addresses associated with the primary interface on that storage device. So, it basically "works" because the XenServers see all the addresses being broadcast back. With the initial request coming from a single host and the information not being returned for the other IP addresses to the other pool members (because of no general broadcast), there is therefore no mechanism that triggers the discovery on the other XenServers within the pool.

So, as mentioned before, one option would be to look if multiple addresses are somehow specified on the initial connection and use that to trigger pushing the connection request to go to the other hosts in the pool. This is in essence what the sr-repair operation accomplishes, namely it looks to see if there is a PBD associated with the SR on each host within the pool. If so, it tries to plug it in. If not, it will issue a pbd-create command pointing to the specific SR and then try to plug in the PBD. So, in essence, you'd want to trigger that SR repair sequence automatically given this situation. That would still evidently require a modification to the SR creation wizard.

-=Tobias

On 2/16/2015 5:24 PM, Vinícius Ferrão wrote:
Tobias,

I've done more tests about the SR creation over XenCenter.

All the process definitely occurs only on the pool master. Even if the specified networks are not part of the networks of the host. The connection is tried on the default route.

I requested a new Software iSCSI SR with the following address:
192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1

The target IQN is discovered without any problem, since the iSCSI Storage reports the LUNs on any interface.

But when trying to discover the LUN, the process only is done on the pool master. During the discovery process, I left a watch command on both XS to with "netstat -an | grep 192.168" and the results are:

On the pool master, there's a lot of things going on:
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 192.168.10.2:55870 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52654 192.168.11.1:3260 TIME_WAIT
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.11.2:52656 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*

During the discovery, look at the 192.168.21.1 address being searched through the default route:
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
******************
tcp 0 1 10.7.0.101:56481 192.168.21.1:3260 SYN_SENT
******************
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*

Even addresses that does not exist on the pool (since the IQN discovery process reports them):
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
******************
tcp 0 1 10.7.0.101:52787 192.168.30.1:3260 SYN_SENT
******************
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*

If you're wondering the 192.168.30.1 is reported because in the future we will add another XS host to the pool. The point here is that XenCenter should not lookup for this network since it isn't specified during the Software iSCSI creation. On my understanding it should only lookup for the specified addresses.

And to finish the tests, the second XS host remains equally during all the SR creation phase:
[***@xenserver2 ~]# netstat -an | grep -i 192.168
tcp 0 48 10.7.0.102:22 192.168.172.1:58046 ESTABLISHED
udp 0 0 192.168.21.2:123 0.0.0.0:*
udp 0 0 192.168.20.2:123 0.0.0.0:*

As you said, perhaps propagating the initial connection through the host will solve the problem without modifying the XenCenter interface. A more sophisticated approach would be query the network-list over XAPI, retrieve the PIFs associated to those networks and them just check the IP addresses on the SR creation request and match them with the equivalent PIFs.

For example here is the output of the xe network-param-list to retrieve the associated PIF's and the xe pif-param-lst with the UUID's of the retrieved PIFs:

[***@xenserver1 ~]# xe network-param-list uuid=4dc936d7-e806-c69a-a292-78e0ed2d5faa
uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa
name-label ( RW): iSCSI #0
name-description ( RW):
VIF-uuids (SRO):
PIF-uuids (SRO): 3baa8436-feca-cd04-3173-5002d9ffc66b; 362d63cb-647a-465f-6b81-1b59cd3b1f5d
MTU ( RW): 9000
bridge ( RO): xenbr2
other-config (MRW): automatic: true
blobs ( RO):
tags (SRW):
default-locking-mode ( RW): unlocked


[***@xenserver1 ~]# xe pif-param-list uuid=3baa8436-feca-cd04-3173-5002d9ffc66b
uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5c:64
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-master-of ( RO):
bond-slave-of ( RO): <not in database>
tunnel-access-PIF-of ( RO):
tunnel-transport-PIF-of ( RO):
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
host-name-label ( RO): xenserver2.local.iq.ufrj.br<http://xenserver2.local.iq.ufrj.br/>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.20.2
netmask ( RO): 255.255.255.252
gateway ( RO):
IPv6-configuration-mode ( RO): None
IPv6 ( RO):
IPv6-gateway ( RO):
primary-address-type ( RO): IPv4
DNS ( RO):
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0


[***@xenserver1 ~]# xe pif-param-list uuid=362d63cb-647a-465f-6b81-1b59cd3b1f5d
uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5a:1c
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-master-of ( RO):
bond-slave-of ( RO): <not in database>
tunnel-access-PIF-of ( RO):
tunnel-transport-PIF-of ( RO):
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c
host-name-label ( RO): xenserver1.local.iq.ufrj.br<http://xenserver1.local.iq.ufrj.br/>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.10.2
netmask ( RO): 255.255.255.252
gateway ( RO):
IPv6-configuration-mode ( RO): None
IPv6 ( RO):
IPv6-gateway ( RO):
primary-address-type ( RO): IPv4
DNS ( RO):
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0

So as you can see, we have the IP addresses and the network mask. Which is sufficient to an efficient and precise algorithm of SR creation.

I thinks this clarifies everything we discussed until now. A patch on XenCenter appears to be really viable.

Many thanks,
Vinicius.

On Feb 16, 2015, at 9:02 PM, Tobias Kreidl <***@nau.edu<mailto:***@nau.edu>> wrote:

Yes, correct. The MD36xx looks like two separate controller units, each acting as a separate device. For a standalone server or some storage devices, that's right that you need a separate subnet for each connection.

Because a single connection isn't seeing the broadcast from the other connectors at the time of the initial connection from one host, the pool is unaware of the others. The "repair" operation causes this to be conducted from each host, and as I guessed, then fixes the SR connectivity for the entire pool. I am speculating that the request for a new connection of this type via XenCenter could perhaps be solved by propagating that initial connection process to the rest of the hosts in the pool and then rescan the SR.

-=Tobias

On 2/16/2015 2:20 PM, Vinícius Ferrão wrote:

Hi Tobias,

As for I know, the Dell MD36xx is a dual controller based storage. So it's basically "two computers". Considering this with the added knowledge that I've about TCP networking there's a golden rule that says: only one IP address on a given subnet on a single computer. So I can't use the same network on my case. That's why I use point-to-point links too.

Since I've only one machine acting as a storage server (It's a Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do need four separate networks. FreeNAS is FreeBSD based, and BSD enforces this rule by default. I can't have two IP addresses of the same subnet on same machine, even if I manually configure it.

If this wasn't bad TCP network practice, I would be able to configure the SR via XenCenter, since it will "find" the same network on the others hosts without problem.

About the XenCenter, it really appears to be only a missing function on the software. If I was a good programmer I would try to implement this. But this is beyond my knowledge... :(

Thanks once again,
Vinícius.


On Feb 16, 2015, at 6:53 PM, Tobias Kreidl <***@nau.edu<mailto:***@nau.edu>> wrote:

No reason to necessarily use four separate subnets. You could have 192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and 2.2 on the other controller (on, for example, an MD36xx that is standard). If however it is something like an MD32xx which recommends four separate subets, then of course, use four. One should follow the vendor's specifications.

I will respond in more detail later when I get a chance, Vinícius, but after a quick read, I agree with your findings from your other email. It would appear that the initial connection is indeed evidently a missing function in XenCenter and perhaps all that is needed to make this work "out of the box". As to the wildcard discovery, it's generally preferred as it allows the host to figure out the path on its own. The only reason I could see not using a wildcard discovery would be if there were any chance for overlaps and ambiguous connections.

Regards,
--Tobias


On 2/16/2015 1:38 PM, Vinícius Ferrão wrote:

Hello Dave,

In total I have four paths. Two in each host. I've made a drawning using ASCII characters, hope this can clarify the architecture:

+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+

The Storage has four gigabit ethernet adapters with the following IP addresses:

192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30

On the first XenServer there are two interfaces with those IP's:

192.168.10.2/30
192.168.11.2/30

And on the second host the equivalent configuration:

192.168.20.2/30
192.168.21.2/30

That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.

To exemplify a little more heres a screenshot of the network configuration made on XenCenter:
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0

Cheers,
Vinícius.


On Feb 15, 2015, at 12:22 PM, Dave Scott <***@citrix.com><mailto:***@citrix.com> wrote:

Hi,


On 15 Feb 2015, at 05:44, Vinícius Ferrão <***@ferrao.eti.br><mailto:***@ferrao.eti.br> wrote:

Hello Tobias,
Thanks for the reply, even to discourage me :)

I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.

Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.

Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked
 Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.

Leaving all those “political” points aside, let’s talk technically :)

I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0

I’m really excited because it worked. So let’s me explain what I’ve did.

First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.

Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.

That’s when things got interesting. To create the PBD, I do need the iSCSI working in the host, and consequently the multipath connection. So I done the process in the terminal with the following commands:

#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1

#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login

#Enable Multipath
multipath
multipath -ll

After all I was able to create the PBD with those new settings. The tricky part was to pass the required “device-config” values via the xe pbd-create command. I wasn’t aware of the method, but after 4 tries I understood the process and typed the following command:

#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I’d like to understand your configuration better— have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?

Thanks,
Dave


Basically everything necessary is in there. The IP’s, the name, the multipath link, the LUN values, everything. After this, I just plugged the PDB via the CLI:

#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342

And BOOM! It worked. As you can see in the screenshot.

To prove that’s everything is working fine, I’ve created a new VM, installed FreeBSD 10 on it, done a pkg install xe-guest-utilities and requested a XenMotion operation. Oh boy, it worked:

Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms

Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0

I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.

What I ask now: a feedback, if possible.

Thanks in advance,
Vinícius.

PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.


On Feb 14, 2015, at 8:14 PM, Tobias Kreidl <***@nau.edu><mailto:***@nau.edu> wrote:

Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.

The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.

That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.

Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.

Regards,
-=Tobias

On 2/14/2015 2:26 PM, Vinícius Ferrão wrote:

Hello guys,

I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.

At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.

I’ve looked for help on ServerFault and FreeNAS Forums, since I’m running FreeNAS as iSCSI service. The discussion is going on those links:

http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/

It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.

If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.

Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)

Thanks in advance,
Vinicius.
Germano Percossi
2015-02-18 12:12:11 UTC
Permalink
Yep,

XenCenter can do very little about it but I did not
get this scenario, too.
By the way changes are usually done in sm code even though
here it seems that there is network topology involved.
If this is the case, not even sm is the right place to look at.

Germano
I admit I haven’t fully understood the whole scenario here, but I’m not
sure that XenCenter would be where it would have to be changed. It seems
to me that it would probably be in the SR creation code on the server side.
--
Stephen Turner
Tobias Kreidl
2015-02-18 13:51:06 UTC
Permalink
So, what would be -- the SR creation code?

The fact that the issue could be mitigated via an SR repair procedure
issued from XenCenter
was the motivational factor.

The problem is that a VBD is not created from each host in the absence
of there being a network
switch that is apparently responsible for properly broadcasting the
iSCSI connection to all hosts in
the pool. The question was then how do you make the iSCSI connector
information known to other hosts
if there is no network switch involved when the SR creation is initiated?
Post by Germano Percossi
Yep,
XenCenter can do very little about it but I did not
get this scenario, too.
By the way changes are usually done in sm code even though
here it seems that there is network topology involved.
If this is the case, not even sm is the right place to look at.
Germano
I admit I haven’t fully understood the whole scenario here, but I’m not
sure that XenCenter would be where it would have to be changed. It seems
to me that it would probably be in the SR creation code on the server side.
--
Stephen Turner
Germano Percossi
2015-02-18 15:39:18 UTC
Permalink
VBD or PBD?
Post by Tobias Kreidl
So, what would be -- the SR creation code?
The fact that the issue could be mitigated via an SR repair procedure
issued from XenCenter
was the motivational factor.
The problem is that a VBD is not created from each host in the absence
of there being a network
Tobias Kreidl
2015-02-18 15:41:00 UTC
Permalink
PBD, of course (sorry -- not enough coffee yet!)
Post by Germano Percossi
VBD or PBD?
Post by Tobias Kreidl
So, what would be -- the SR creation code?
The fact that the issue could be mitigated via an SR repair procedure
issued from XenCenter
was the motivational factor.
The problem is that a VBD is not created from each host in the absence
of there being a network
Tim Mackey
2015-02-18 16:36:00 UTC
Permalink
I’ll add in a few more tests
.


1. Host restart. Does the normal PBD plug mechanism work with such a configuration

2. HA. While HA with two hosts isn’t a supported configuration, in this model it might work properly. If you remove the network cables for a single host, does it fence, or do both hosts fence?

3. Multiple SRs. If you have multiple LUNs on the FreeNAS exported and then used as independent SRs, does everything work as it should?

4. Storage XenMotion. With a vdi on one SR presented with the FreeNAS, copy it to any another storage option

5. Motion during path failure. With one path down on a given host, does XenMotion and Storage XenMotion still function correctly?

For those who are questioning the value of such a configuration, this is in essence a “cluster in a box” solution. These configurations were very popular 10-15 years ago as highly available compute infrastructure was quite expensive back then. Today we can accomplish the same things we did back then using virtualization, but “cluster in a box” can always be valuable for those with limited IT skills or for smaller organizations wanting least cost with fewest points of failure.

-tim

From: xs-devel-***@lists.xenserver.org [mailto:xs-devel-***@lists.xenserver.org] On Behalf Of Vinícius Ferrão
Sent: Monday, February 16, 2015 7:24 PM
To: Tobias Kreidl
Cc: xs-***@lists.xenserver.org
Subject: Re: [xs-devel] Switchless shared iSCSI Network with /30 links

Tobias,

I've done more tests about the SR creation over XenCenter.

All the process definitely occurs only on the pool master. Even if the specified networks are not part of the networks of the host. The connection is tried on the default route.

I requested a new Software iSCSI SR with the following address:
192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1

The target IQN is discovered without any problem, since the iSCSI Storage reports the LUNs on any interface.

But when trying to discover the LUN, the process only is done on the pool master. During the discovery process, I left a watch command on both XS to with "netstat -an | grep 192.168" and the results are:

On the pool master, there's a lot of things going on:
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 192.168.10.2:55870 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52654 192.168.11.1:3260 TIME_WAIT
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.11.2:52656 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*

During the discovery, look at the 192.168.21.1 address being searched through the default route:
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
******************
tcp 0 1 10.7.0.101:56481 192.168.21.1:3260 SYN_SENT
******************
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*

Even addresses that does not exist on the pool (since the IQN discovery process reports them):
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
******************
tcp 0 1 10.7.0.101:52787 192.168.30.1:3260 SYN_SENT
******************
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*

If you're wondering the 192.168.30.1 is reported because in the future we will add another XS host to the pool. The point here is that XenCenter should not lookup for this network since it isn't specified during the Software iSCSI creation. On my understanding it should only lookup for the specified addresses.

And to finish the tests, the second XS host remains equally during all the SR creation phase:
[***@xenserver2 ~]# netstat -an | grep -i 192.168
tcp 0 48 10.7.0.102:22 192.168.172.1:58046 ESTABLISHED
udp 0 0 192.168.21.2:123 0.0.0.0:*
udp 0 0 192.168.20.2:123 0.0.0.0:*

As you said, perhaps propagating the initial connection through the host will solve the problem without modifying the XenCenter interface. A more sophisticated approach would be query the network-list over XAPI, retrieve the PIFs associated to those networks and them just check the IP addresses on the SR creation request and match them with the equivalent PIFs.

For example here is the output of the xe network-param-list to retrieve the associated PIF's and the xe pif-param-lst with the UUID's of the retrieved PIFs:

[***@xenserver1 ~]# xe network-param-list uuid=4dc936d7-e806-c69a-a292-78e0ed2d5faa
uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa
name-label ( RW): iSCSI #0
name-description ( RW):
VIF-uuids (SRO):
PIF-uuids (SRO): 3baa8436-feca-cd04-3173-5002d9ffc66b; 362d63cb-647a-465f-6b81-1b59cd3b1f5d
MTU ( RW): 9000
bridge ( RO): xenbr2
other-config (MRW): automatic: true
blobs ( RO):
tags (SRW):
default-locking-mode ( RW): unlocked


[***@xenserver1 ~]# xe pif-param-list uuid=3baa8436-feca-cd04-3173-5002d9ffc66b
uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5c:64
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-master-of ( RO):
bond-slave-of ( RO): <not in database>
tunnel-access-PIF-of ( RO):
tunnel-transport-PIF-of ( RO):
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
host-name-label ( RO): xenserver2.local.iq.ufrj.br<http://xenserver2.local.iq.ufrj.br>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.20.2
netmask ( RO): 255.255.255.252
gateway ( RO):
IPv6-configuration-mode ( RO): None
IPv6 ( RO):
IPv6-gateway ( RO):
primary-address-type ( RO): IPv4
DNS ( RO):
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0


[***@xenserver1 ~]# xe pif-param-list uuid=362d63cb-647a-465f-6b81-1b59cd3b1f5d
uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5a:1c
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-master-of ( RO):
bond-slave-of ( RO): <not in database>
tunnel-access-PIF-of ( RO):
tunnel-transport-PIF-of ( RO):
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c
host-name-label ( RO): xenserver1.local.iq.ufrj.br<http://xenserver1.local.iq.ufrj.br>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.10.2
netmask ( RO): 255.255.255.252
gateway ( RO):
IPv6-configuration-mode ( RO): None
IPv6 ( RO):
IPv6-gateway ( RO):
primary-address-type ( RO): IPv4
DNS ( RO):
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0

So as you can see, we have the IP addresses and the network mask. Which is sufficient to an efficient and precise algorithm of SR creation.

I thinks this clarifies everything we discussed until now. A patch on XenCenter appears to be really viable.

Many thanks,
Vinicius.

On Feb 16, 2015, at 9:02 PM, Tobias Kreidl <***@nau.edu<mailto:***@nau.edu>> wrote:

Yes, correct. The MD36xx looks like two separate controller units, each acting as a separate device. For a standalone server or some storage devices, that's right that you need a separate subnet for each connection.

Because a single connection isn't seeing the broadcast from the other connectors at the time of the initial connection from one host, the pool is unaware of the others. The "repair" operation causes this to be conducted from each host, and as I guessed, then fixes the SR connectivity for the entire pool. I am speculating that the request for a new connection of this type via XenCenter could perhaps be solved by propagating that initial connection process to the rest of the hosts in the pool and then rescan the SR.

-=Tobias

On 2/16/2015 2:20 PM, Vinícius Ferrão wrote:

Hi Tobias,

As for I know, the Dell MD36xx is a dual controller based storage. So it's basically "two computers". Considering this with the added knowledge that I've about TCP networking there's a golden rule that says: only one IP address on a given subnet on a single computer. So I can't use the same network on my case. That's why I use point-to-point links too.

Since I've only one machine acting as a storage server (It's a Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do need four separate networks. FreeNAS is FreeBSD based, and BSD enforces this rule by default. I can't have two IP addresses of the same subnet on same machine, even if I manually configure it.

If this wasn't bad TCP network practice, I would be able to configure the SR via XenCenter, since it will "find" the same network on the others hosts without problem.

About the XenCenter, it really appears to be only a missing function on the software. If I was a good programmer I would try to implement this. But this is beyond my knowledge... :(

Thanks once again,
Vinícius.


On Feb 16, 2015, at 6:53 PM, Tobias Kreidl <***@nau.edu<mailto:***@nau.edu>> wrote:

No reason to necessarily use four separate subnets. You could have 192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and 2.2 on the other controller (on, for example, an MD36xx that is standard). If however it is something like an MD32xx which recommends four separate subets, then of course, use four. One should follow the vendor's specifications.

I will respond in more detail later when I get a chance, Vinícius, but after a quick read, I agree with your findings from your other email. It would appear that the initial connection is indeed evidently a missing function in XenCenter and perhaps all that is needed to make this work "out of the box". As to the wildcard discovery, it's generally preferred as it allows the host to figure out the path on its own. The only reason I could see not using a wildcard discovery would be if there were any chance for overlaps and ambiguous connections.

Regards,
--Tobias


On 2/16/2015 1:38 PM, Vinícius Ferrão wrote:

Hello Dave,

In total I have four paths. Two in each host. I've made a drawning using ASCII characters, hope this can clarify the architecture:

+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+

The Storage has four gigabit ethernet adapters with the following IP addresses:

192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30

On the first XenServer there are two interfaces with those IP's:

192.168.10.2/30
192.168.11.2/30

And on the second host the equivalent configuration:

192.168.20.2/30
192.168.21.2/30

That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.

To exemplify a little more heres a screenshot of the network configuration made on XenCenter:
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0

Cheers,
Vinícius.


On Feb 15, 2015, at 12:22 PM, Dave Scott <***@citrix.com<mailto:***@citrix.com>> wrote:

Hi,


On 15 Feb 2015, at 05:44, Vinícius Ferrão <***@ferrao.eti.br<mailto:***@ferrao.eti.br>> wrote:

Hello Tobias,
Thanks for the reply, even to discourage me :)

I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.

Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.

Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked
 Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.

Leaving all those “political” points aside, let’s talk technically :)

I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0

I’m really excited because it worked. So let’s me explain what I’ve did.

First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.

Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.

That’s when things got interesting. To create the PBD, I do need the iSCSI working in the host, and consequently the multipath connection. So I done the process in the terminal with the following commands:

#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1

#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login

#Enable Multipath
multipath
multipath -ll

After all I was able to create the PBD with those new settings. The tricky part was to pass the required “device-config” values via the xe pbd-create command. I wasn’t aware of the method, but after 4 tries I understood the process and typed the following command:

#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I’d like to understand your configuration better— have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?

Thanks,
Dave


Basically everything necessary is in there. The IP’s, the name, the multipath link, the LUN values, everything. After this, I just plugged the PDB via the CLI:

#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342

And BOOM! It worked. As you can see in the screenshot.

To prove that’s everything is working fine, I’ve created a new VM, installed FreeBSD 10 on it, done a pkg install xe-guest-utilities and requested a XenMotion operation. Oh boy, it worked:

Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms

Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0

I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.

What I ask now: a feedback, if possible.

Thanks in advance,
Vinícius.

PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.


On Feb 14, 2015, at 8:14 PM, Tobias Kreidl <***@nau.edu<mailto:***@nau.edu>> wrote:

Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.

The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.

That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.

Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.

Regards,
-=Tobias

On 2/14/2015 2:26 PM, Vinícius Ferrão wrote:

Hello guys,

I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.

At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.

I’ve looked for help on ServerFault and FreeNAS Forums, since I’m running FreeNAS as iSCSI service. The discussion is going on those links:

http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/

It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.

If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.

Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)

Thanks in advance,
Vinicius.
Tobias Kreidl
2015-02-18 17:26:56 UTC
Permalink
Good set of checks, Tim.

So, again, is it the SR creation code that is then responsible for
properly establishing all the connections, as it should recognize that
PBDs need to be created and plugged in for each member of the pool?
Anyone know off-hand what module that might be contained in?

XenCenter may also come into play because if you feed it multiple IP
addresses, whatever parses that and passes it on to the underlying
process would have to then be able to recognize that this is a "special"
case and require possible slightly different processing, right? If,
however, all the logic lies within that process then XenCenter would
need no changes and the process itself would recognize the multiple IP
addresses and need to trigger the proper next steps accordingly.

While not initially enthusiastic about this, my curiosity has been
re-awakened in particular because someone in another department has put
off going to 10 Gb NICs because of the high cost of network switches,
while a dual 10 Gb NIC can now be had for around $500. Hence, this is
indeed an interesting and potentially viable alternative.

-=Tobias
Post by Tim Mackey
I’ll add in a few more tests
.
1.Host restart. Does the normal PBD plug mechanism work with such a
configuration
2.HA. While HA with two hosts isn’t a supported configuration, in this
model it might work properly. If you remove the network cables for a
single host, does it fence, or do both hosts fence?
3.Multiple SRs. If you have multiple LUNs on the FreeNAS exported and
then used as independent SRs, does everything work as it should?
4.Storage XenMotion. With a vdi on one SR presented with the FreeNAS,
copy it to any another storage option
5.Motion during path failure. With one path down on a given host,
does XenMotion and Storage XenMotion still function correctly?
For those who are questioning the value of such a configuration, this
is in essence a “cluster in a box” solution. These configurations
were very popular 10-15 years ago as highly available compute
infrastructure was quite expensive back then. Today we can accomplish
the same things we did back then using virtualization, but “cluster in
a box” can always be valuable for those with limited IT skills or for
smaller organizations wanting least cost with fewest points of failure.
-tim
Ferrão
*Sent:* Monday, February 16, 2015 7:24 PM
*To:* Tobias Kreidl
*Subject:* Re: [xs-devel] Switchless shared iSCSI Network with /30 links
Tobias,
I've done more tests about the SR creation over XenCenter.
All the process definitely occurs only on the pool master. Even if the
specified networks are not part of the networks of the host. The
connection is tried on the default route.
192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1
The target IQN is discovered without any problem, since the iSCSI
Storage reports the LUNs on any interface.
But when trying to discover the LUN, the process only is done on the
pool master. During the discovery process, I left a watch command on
tcp 0 0 10.7.0.101:443 192.168.172.1:58053
ESTABLISHED
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365
ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058
ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266
ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930
ESTABLISHED
tcp 0 48 10.7.0.101:22 192.168.172.1:58064
ESTABLISHED
tcp 0 0 192.168.10.2:55870 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:52654 192.168.11.1:3260
TIME_WAIT
tcp 0 0 10.7.0.101:443 192.168.172.1:58055
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054
ESTABLISHED
tcp 0 0 192.168.11.2:52656 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260
TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
During the discovery, look at the 192.168.21.1 address being searched
tcp 0 0 10.7.0.101:443 192.168.172.1:58053
ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365
ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058
ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266
ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930
ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260
TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064
ESTABLISHED
******************
tcp 0 1 10.7.0.101:56481 192.168.21.1:3260
SYN_SENT
******************
tcp 0 0 10.7.0.101:443 192.168.172.1:58055
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054
ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260
TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
Even addresses that does not exist on the pool (since the IQN
tcp 0 0 10.7.0.101:443 192.168.172.1:58053
ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365
ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058
ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266
ESTABLISHED
******************
tcp 0 1 10.7.0.101:52787 192.168.30.1:3260
SYN_SENT
******************
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930
ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260
TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58055
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054
ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260
TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
If you're wondering the 192.168.30.1 is reported because in the future
we will add another XS host to the pool. The point here is that
XenCenter should not lookup for this network since it isn't specified
during the Software iSCSI creation. On my understanding it should only
lookup for the specified addresses.
And to finish the tests, the second XS host remains equally during all
tcp 0 48 10.7.0.102:22 192.168.172.1:58046
ESTABLISHED
udp 0 0 192.168.21.2:123 0.0.0.0:*
udp 0 0 192.168.20.2:123 0.0.0.0:*
As you said, perhaps propagating the initial connection through the
host will solve the problem without modifying the XenCenter interface.
A more sophisticated approach would be query the network-list over
XAPI, retrieve the PIFs associated to those networks and them just
check the IP addresses on the SR creation request and match them with
the equivalent PIFs.
For example here is the output of the xe network-param-list to
retrieve the associated PIF's and the xe pif-param-lst with the UUID's
uuid=4dc936d7-e806-c69a-a292-78e0ed2d5faa
uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa
name-label ( RW): iSCSI #0
PIF-uuids (SRO): 3baa8436-feca-cd04-3173-5002d9ffc66b;
362d63cb-647a-465f-6b81-1b59cd3b1f5d
MTU ( RW): 9000
bridge ( RO): xenbr2
other-config (MRW): automatic: true
default-locking-mode ( RW): unlocked
uuid=3baa8436-feca-cd04-3173-5002d9ffc66b
uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5c:64
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
host-name-label ( RO): xenserver2.local.iq.ufrj.br
<http://xenserver2.local.iq.ufrj.br>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.20.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
uuid=362d63cb-647a-465f-6b81-1b59cd3b1f5d
uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5a:1c
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c
host-name-label ( RO): xenserver1.local.iq.ufrj.br
<http://xenserver1.local.iq.ufrj.br>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.10.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
So as you can see, we have the IP addresses and the network mask.
Which is sufficient to an efficient and precise algorithm of SR creation.
I thinks this clarifies everything we discussed until now. A patch on
XenCenter appears to be really viable.
Many thanks,
Vinicius.
Yes, correct. The MD36xx looks like two separate controller units,
each acting as a separate device. For a standalone server or some
storage devices, that's right that you need a separate subnet for
each connection.
Because a single connection isn't seeing the broadcast from the
other connectors at the time of the initial connection from one
host, the pool is unaware of the others. The "repair" operation
causes this to be conducted from each host, and as I guessed, then
fixes the SR connectivity for the entire pool. I am speculating
that the request for a new connection of this type via XenCenter
could perhaps be solved by propagating that initial connection
process to the rest of the hosts in the pool and then rescan the SR.
-=Tobias
Hi Tobias,
As for I know, the Dell MD36xx is a dual controller based storage.
So it's basically "two computers". Considering this with the added
knowledge that I've about TCP networking there's a golden rule
that says: only one IP address on a given subnet on a single
computer. So I can't use the same network on my case. That's why I
use point-to-point links too.
Since I've only one machine acting as a storage server (It's a
Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do
need four separate networks. FreeNAS is FreeBSD based, and BSD
enforces this rule by default. I can't have two IP addresses of
the same subnet on same machine, even if I manually configure it.
If this wasn't bad TCP network practice, I would be able to
configure the SR via XenCenter, since it will "find" the same
network on the others hosts without problem.
About the XenCenter, it really appears to be only a missing
function on the software. If I was a good programmer I would try
to implement this. But this is beyond my knowledge... :(
Thanks once again,
Vinícius.
No reason to necessarily use four separate subnets. You could
have 192.168.10.1. and 20.1 on one iSCSI controller and
192.168.10.2 and 2.2 on the other controller (on, for example, an
MD36xx that is standard). If however it is something like an
MD32xx which recommends four separate subets, then of course, use
four. One should follow the vendor's specifications.
I will respond in more detail later when I get a chance, Vinícius,
but after a quick read, I agree with your findings from your other
email. It would appear that the initial connection is indeed
evidently a missing function in XenCenter and perhaps all that is
needed to make this work "out of the box". As to the wildcard
discovery, it's generally preferred as it allows the host to
figure out the path on its own. The only reason I could see not
using a wildcard discovery would be if there were any chance for
overlaps and ambiguous connections.
Regards,
--Tobias
Hello Dave,
In total I have four paths. Two in each host. I've made a drawning
+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+
192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30
192.168.10.2/30
192.168.11.2/30
192.168.20.2/30
192.168.21.2/30
That's why I'm using multipath. Because I do need MPIO to sum the
two network interfaces per host.
To exemplify a little more heres a screenshot of the network
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0
Cheers,
Vinícius.
Hi,
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it,
they recommend, and they even sell a solution using DELL hardware
with two machines and a shared storage running with 10GbE links.
Point-ot-point networks, dual controllers on the storage side and
two different network for iSCSI. They sell it at least here in
Brazil. What I’m trying to do is achieve the same topology with
XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a
better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different
markets. For example, I’ve used Fibre Channel only one time in my
life. And, believe it or not, it was a switchless configuration.
And it worked
 Wasn’t for VM workload but was in a GRID Cluster
with TORQUE/OpenPBS software. I do have the pictures of the
topology, but I need to search for it. At that time, I wasn’t the
guy who created the solution, but I understood that it was a huge
money saving, because the FC switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was
necessary in the CLI, but I was comfortable with this. The
https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0
I’m really excited because it worked. So let’s me explain what
I’ve did.
First I started creating the SR via XenCenter, it failed as
expected. But to bypass I just created the SR via XenCenter with
the two Storage IP’s of the Pool Master. So the process went OK,
but it started broken, since the second XS host cannot see the IP
addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR
UUID created by XenCenter and the referenced PBDs. According to
what I know about XS, the PBD’s are the physical connect from
separate XS hosts to the shared storage. So it’s not related to
other hosts, it’s exclusively to the host machine that I’m using.
Understood that, the ideia is to destroy the created PBD on the
second XS host and recreate it with the proper settings and IP
addresses.
That’s when things got interesting. To create the PBD, I do need
the iSCSI working in the host, and consequently the multipath
connection. So I done the process in the terminal with the
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p
192.168.20.1:3260 --login
iscsiadm -m node --targetname
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p
192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
After all I was able to create the PBD with those new settings.
The tricky part was to pass the required “device-config” values
via the xe pbd-create command. I wasn’t aware of the method, but
after 4 tries I understood the process and typed the following
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f
host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|"
device-config:target=192.168.20.1,192.168.21.1
device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260
device-config:targetIQN=*
device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000
device-config:port=3260
I’d like to understand your configuration better— have you
effectively enabled 2 storage paths for the SR, knowing that each
host will only be able to use one path? Are your PBDs identical or
are there differences?
Thanks,
Dave
Basically everything necessary is in there. The IP’s, the name,
the multipath link, the LUN values, everything. After this, I just
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
To prove that’s everything is working fine, I’ve created a new VM,
installed FreeBSD 10 on it, done a pkg install xe-guest-utilities
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0
I thinks this is sufficient for now, to prove that the concept
works and XenServer can create this kind of topology. It’s just
not available on the GUI. If XenCenter supports this kind of SR on
the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was
really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to
try to squeeze something out of a technology that is not built-in
or future-safe. Would you want to try to connect a fiber-channel
storage device without a fiber channel switch (a very similar
situation)? There are other alternatives, for example, to connect
iSCSi storage to another, XenServer-independent server and then
serve storage from it via NFS or use NFS in the first place and
not make use at all of iSCSI technology. NFS has come a long way
and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are
a significant consideration and cannot be ignored as part of the
storage support protocols within XenServer. XenServer provides a
number of options for storage connectivity to cover a wide range
of needs, preferences and costs. Often, trying to save money in
the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism
altogether (obviously, not a supported configuration) and connect
at least Linux VMs directly to iSCSI storage using open iSCSI, but
I have not explored this without still involving a network switch.
As you indicated, it's also not clear if XenServer will remain
100% open iSCSI compatible or not in the future. It also takes
aways some of the things an SR can offer, like VM cloning, storage
migration, etc.
Finally, I am not sure I see where the lack of a switch factors in
in improving performance. Networks switches are much more capable
and much faster than any storage device or host in routing
packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of
using a iSCSI SR without a Switch. The main goal is to reduce the
TCO of the solution, get some extra performance removing the
overhead that a managed switch puts on the network and reduce the
complexity of the network.
At a first glance I thought that I will only achieve those kind of
setup with a distributed switch, so DVSC should be an option. But
as Tim said it’s only a interface to control some policies.
I’ve looked for help on ServerFault and FreeNAS Forums, since I’m
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/
It appears that XenServer cannot support switchless iSCSI storages
without a switch, because XS needs the same network block on all
the XS hosts to create a shared storage between the hosts. So the
ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate
PBD’s. Perhaps one way to solve this issue is ignoring XenCenter
on SR creation and try to create an SR with all the PBD’s from the
XS hosts. But I was unable to test this setting due to lack of my
knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of
the developers, to see if this method of building the iSCSI
network is viable, since as Tim said, XS is not 100% open-iscsi
upstream and if it could be integrated on next versions of
XenServer via the XenCenter GUI. I think that’s a very common
architecture and VMware is doing this on small pools! So let’s do
it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Vinícius Ferrão
2015-02-18 17:47:36 UTC
Permalink
Hello guys,

Answering about all the questions and issues. For those who aren't understanding the network topology, I've made a drawing of the solution. The image is here: Loading Image... <Loading Image...

In the drawing you can see the iSCSI Network made with point-to-point links. The IP's on the storage box are:
192.168.10.1
192.168.11.1
192.168.20.1
192.168.21.1

The convention to this numbering scheme was: 192.168.{XenServerNumber}{InterfaceNumber}.{Storage(1)/Host(2)}.

So on the host #1 I've two addresses and more two on host #2:
192.168.10.2 (XenServer #1 - Interface #0)
192.168.11.2 (XenServer #1 - Interface #1)
192.168.20.2 (XenServer #2 - Interface #0)
192.168.21.2 (XenServer #2 - Interface #1)

Hope this completely clarifies the architecture/topology of the iSCSI network.

Talking about the tests raised by Tim:

1. Already done this test. Tried three different tests:
Restarting both hosts on the same time. [OK]
Restarting the second host. [OK]
Restarting the pool master. [OK]

2. HA is working too. It fences only the damaged host. I even tried a VM Failover. HA happened without any problem and the lost VM has been restarted on other server. To do the HA test, I've power cycled the host, simulated a hardware crash scenario. A cool picture: Loading Image... <Loading Image...

3. I haven't understood this test. You are asking to create a new LUN and add a new Shared Repository? It's not clear what you're saying with "independent SR".

4. Storage XenMotion works too. I've moved a running VM from local storage (on the Host #2) to the iSCSI pool without any problem. The reverse process works too.

5. I will test this tomorrow, when I will got physical access to the servers (carnival holiday here in BR). But about this specific test, I can't see any problem with plain Xen Motion, since the data is copied over the management network and not the iSCSI network. But I can see a good point when dealing with Storage XenMotion.

Finally about the changes on XenCenter or XenServer/XAPI itself. One thing to prove this point is to create the SR via the CLI. If I would be able to do this, the problem should definitely be on XenCenter. As I said before, I've created a broken Software iSCSI SR via XenCenter and them mapped the iSCSI interfaces, enabled multipath and created the PBD over the CLI.

The iSCSI and Multipath should be done in the physical host. But the PBD creation can be done without any problem on the Pool Master.

To test this scenario I do require some help crafting the xe sr-create command, because I don't know exactly how this command is formed by XenCenter, and it requires some doubtful inputs:

[***@xenserver1 ~]# xe sr-create
content-type= host-uuid= physical-size= sm-config:
device-config: name-label= shared= type=

In this case sm-config and device-config are the problematic ones. Assuming that the host-uuid is always the pool master.

Thanks in advance,
Vinícius.
Post by Tim Mackey
I’ll add in a few more tests
.
1. Host restart. Does the normal PBD plug mechanism work with such a configuration
2. HA. While HA with two hosts isn’t a supported configuration, in this model it might work properly. If you remove the network cables for a single host, does it fence, or do both hosts fence?
3. Multiple SRs. If you have multiple LUNs on the FreeNAS exported and then used as independent SRs, does everything work as it should?
4. Storage XenMotion. With a vdi on one SR presented with the FreeNAS, copy it to any another storage option
5. Motion during path failure. With one path down on a given host, does XenMotion and Storage XenMotion still function correctly?
For those who are questioning the value of such a configuration, this is in essence a “cluster in a box” solution. These configurations were very popular 10-15 years ago as highly available compute infrastructure was quite expensive back then. Today we can accomplish the same things we did back then using virtualization, but “cluster in a box” can always be valuable for those with limited IT skills or for smaller organizations wanting least cost with fewest points of failure.
-tim
Sent: Monday, February 16, 2015 7:24 PM
To: Tobias Kreidl
Subject: Re: [xs-devel] Switchless shared iSCSI Network with /30 links
Tobias,
I've done more tests about the SR creation over XenCenter.
All the process definitely occurs only on the pool master. Even if the specified networks are not part of the networks of the host. The connection is tried on the default route.
192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1
The target IQN is discovered without any problem, since the iSCSI Storage reports the LUNs on any interface.
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 192.168.10.2:55870 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52654 192.168.11.1:3260 TIME_WAIT
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.11.2:52656 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
******************
tcp 0 1 10.7.0.101:56481 192.168.21.1:3260 SYN_SENT
******************
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
******************
tcp 0 1 10.7.0.101:52787 192.168.30.1:3260 SYN_SENT
******************
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
If you're wondering the 192.168.30.1 is reported because in the future we will add another XS host to the pool. The point here is that XenCenter should not lookup for this network since it isn't specified during the Software iSCSI creation. On my understanding it should only lookup for the specified addresses.
tcp 0 48 10.7.0.102:22 192.168.172.1:58046 ESTABLISHED
udp 0 0 192.168.21.2:123 0.0.0.0:*
udp 0 0 192.168.20.2:123 0.0.0.0:*
As you said, perhaps propagating the initial connection through the host will solve the problem without modifying the XenCenter interface. A more sophisticated approach would be query the network-list over XAPI, retrieve the PIFs associated to those networks and them just check the IP addresses on the SR creation request and match them with the equivalent PIFs.
uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa
name-label ( RW): iSCSI #0
PIF-uuids (SRO): 3baa8436-feca-cd04-3173-5002d9ffc66b; 362d63cb-647a-465f-6b81-1b59cd3b1f5d
MTU ( RW): 9000
bridge ( RO): xenbr2
other-config (MRW): automatic: true
default-locking-mode ( RW): unlocked
uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5c:64
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
host-name-label ( RO): xenserver2.local.iq.ufrj.br <http://xenserver2.local.iq.ufrj.br/>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.20.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5a:1c
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c
host-name-label ( RO): xenserver1.local.iq.ufrj.br <http://xenserver1.local.iq.ufrj.br/>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.10.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
So as you can see, we have the IP addresses and the network mask. Which is sufficient to an efficient and precise algorithm of SR creation.
I thinks this clarifies everything we discussed until now. A patch on XenCenter appears to be really viable.
Many thanks,
Vinicius.
Yes, correct. The MD36xx looks like two separate controller units, each acting as a separate device. For a standalone server or some storage devices, that's right that you need a separate subnet for each connection.
Because a single connection isn't seeing the broadcast from the other connectors at the time of the initial connection from one host, the pool is unaware of the others. The "repair" operation causes this to be conducted from each host, and as I guessed, then fixes the SR connectivity for the entire pool. I am speculating that the request for a new connection of this type via XenCenter could perhaps be solved by propagating that initial connection process to the rest of the hosts in the pool and then rescan the SR.
-=Tobias
Hi Tobias,
As for I know, the Dell MD36xx is a dual controller based storage. So it's basically "two computers". Considering this with the added knowledge that I've about TCP networking there's a golden rule that says: only one IP address on a given subnet on a single computer. So I can't use the same network on my case. That's why I use point-to-point links too.
Since I've only one machine acting as a storage server (It's a Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do need four separate networks. FreeNAS is FreeBSD based, and BSD enforces this rule by default. I can't have two IP addresses of the same subnet on same machine, even if I manually configure it.
If this wasn't bad TCP network practice, I would be able to configure the SR via XenCenter, since it will "find" the same network on the others hosts without problem.
About the XenCenter, it really appears to be only a missing function on the software. If I was a good programmer I would try to implement this. But this is beyond my knowledge... :(
Thanks once again,
Vinícius.
No reason to necessarily use four separate subnets. You could have 192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and 2.2 on the other controller (on, for example, an MD36xx that is standard). If however it is something like an MD32xx which recommends four separate subets, then of course, use four. One should follow the vendor's specifications.
I will respond in more detail later when I get a chance, Vinícius, but after a quick read, I agree with your findings from your other email. It would appear that the initial connection is indeed evidently a missing function in XenCenter and perhaps all that is needed to make this work "out of the box". As to the wildcard discovery, it's generally preferred as it allows the host to figure out the path on its own. The only reason I could see not using a wildcard discovery would be if there were any chance for overlaps and ambiguous connections.
Regards,
--Tobias
Hello Dave,
+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+
192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30
192.168.10.2/30
192.168.11.2/30
192.168.20.2/30
192.168.21.2/30
That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0 <https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0>
Cheers,
Vinícius.
Hi,
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked
 Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0 <https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0>
I’m really excited because it worked. So let’s me explain what I’ve did.
First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I’d like to understand your configuration better— have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?
Thanks,
Dave
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0 <https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0>
I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch <http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch>
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/ <https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/>
It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Tobias Kreidl
2015-02-18 18:03:10 UTC
Permalink
Vinícius:

Regarding Point 3) I believe what is meant is that you should test
having more than one SR that you are connecting to using the same IP
addresses for the target array. You know it works fine to create one SR,
so can you add additional ones the same way and have them all live
together in harmony? :-)

As to the sr-create command, the host-uuid is an optional parameter and
hence does not need to be named for the pool master or any other. The
only required parameters are the name-label and the SR type. Have you
tried that to see if the behavior is any different compared to when you
/do/ supply a host name?

-=Tobias
Post by Vinícius Ferrão
Hello guys,
Answering about all the questions and issues. For those who aren't
understanding the network topology, I've made a drawing of the
https://www.dropbox.com/s/6zwerk6igcyqou3/UFRJ-IQ-VirtualPool.jpg?dl=0
In the drawing you can see the iSCSI Network made with point-to-point
192.168.10.1
192.168.11.1
192.168.20.1
192.168.21.1
192.168.{XenServerNumber}{InterfaceNumber}.{Storage(1)/Host(2)}.
192.168.10.2 (XenServer #1 - Interface #0)
192.168.11.2 (XenServer #1 - Interface #1)
192.168.20.2 (XenServer #2 - Interface #0)
192.168.21.2 (XenServer #2 - Interface #1)
Hope this completely clarifies the architecture/topology of the iSCSI network.
Restarting both hosts on the same time. [OK]
Restarting the second host. [OK]
Restarting the pool master. [OK]
2. HA is working too. It fences only the damaged host. I even tried a
VM Failover. HA happened without any problem and the lost VM has been
restarted on other server. To do the HA test, I've power cycled the
https://www.dropbox.com/s/g7h7rljk8vh97rg/Screenshot%202015-02-18%2015.45.41.png?dl=0
3. I haven't understood this test. You are asking to create a new LUN
and add a new Shared Repository? It's not clear what you're saying
with "independent SR".
4. Storage XenMotion works too. I've moved a running VM from local
storage (on the Host #2) to the iSCSI pool without any problem. The
reverse process works too.
5. I will test this tomorrow, when I will got physical access to the
servers (carnival holiday here in BR). But about this specific test, I
can't see any problem with plain Xen Motion, since the data is copied
over the management network and not the iSCSI network. But I can see
a good point when dealing with Storage XenMotion.
Finally about the changes on XenCenter or XenServer/XAPI itself. One
thing to prove this point is to create the SR via the CLI. If I would
be able to do this, the problem should definitely be on XenCenter. As
I said before, I've created a broken Software iSCSI SR via XenCenter
and them mapped the iSCSI interfaces, enabled multipath and created
the PBD over the CLI.
The iSCSI and Multipath should be done in the physical host. But the
PBD creation can be done without any problem on the Pool Master.
To test this scenario I do require some help crafting the xe sr-create
command, because I don't know exactly how this command is formed by
device-config: name-label= shared= type=
In this case sm-config and device-config are the problematic ones.
Assuming that the host-uuid is always the pool master.
Thanks in advance,
Vinícius.
Post by Tim Mackey
I’ll add in a few more tests
.
1.Host restart. Does the normal PBD plug mechanism work with such a
configuration
2.HA. While HA with two hosts isn’t a supported configuration, in
this model it might work properly. If you remove the network cables
for a single host, does it fence, or do both hosts fence?
3.Multiple SRs. If you have multiple LUNs on the FreeNAS exported
and then used as independent SRs, does everything work as it should?
4.Storage XenMotion. With a vdi on one SR presented with the
FreeNAS, copy it to any another storage option
5.Motion during path failure. With one path down on a given host,
does XenMotion and Storage XenMotion still function correctly?
For those who are questioning the value of such a configuration, this
is in essence a “cluster in a box” solution. These configurations
were very popular 10-15 years ago as highly available compute
infrastructure was quite expensive back then. Today we can
accomplish the same things we did back then using virtualization, but
“cluster in a box” can always be valuable for those with limited IT
skills or for smaller organizations wanting least cost with fewest
points of failure.
-tim
Ferrão
*Sent:*Monday, February 16, 2015 7:24 PM
*To:*Tobias Kreidl
*Subject:*Re: [xs-devel] Switchless shared iSCSI Network with /30 links
Tobias,
I've done more tests about the SR creation over XenCenter.
All the process definitely occurs only on the pool master. Even if
the specified networks are not part of the networks of the host. The
connection is tried on the default route.
192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1
The target IQN is discovered without any problem, since the iSCSI
Storage reports the LUNs on any interface.
But when trying to discover the LUN, the process only is done on the
pool master. During the discovery process, I left a watch command on
tcp 0 0 10.7.0.101:443 192.168.172.1:58053
ESTABLISHED
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365
ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058
ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266
ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930
ESTABLISHED
tcp 0 48 10.7.0.101:22 192.168.172.1:58064
ESTABLISHED
tcp 0 0 192.168.10.2:55870 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:52654 192.168.11.1:3260
TIME_WAIT
tcp 0 0 10.7.0.101:443 192.168.172.1:58055
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054
ESTABLISHED
tcp 0 0 192.168.11.2:52656 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260
TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
During the discovery, look at the 192.168.21.1 address being searched
tcp 0 0 10.7.0.101:443 192.168.172.1:58053
ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365
ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058
ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266
ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930
ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260
TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064
ESTABLISHED
******************
tcp 0 1 10.7.0.101:56481 192.168.21.1:3260
SYN_SENT
******************
tcp 0 0 10.7.0.101:443 192.168.172.1:58055
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054
ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260
TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
Even addresses that does not exist on the pool (since the IQN
tcp 0 0 10.7.0.101:443 192.168.172.1:58053
ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365
ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058
ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260
TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266
ESTABLISHED
******************
tcp 0 1 10.7.0.101:52787 192.168.30.1:3260
SYN_SENT
******************
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930
ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260
TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58055
ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054
ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260
TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
If you're wondering the 192.168.30.1 is reported because in the
future we will add another XS host to the pool. The point here is
that XenCenter should not lookup for this network since it isn't
specified during the Software iSCSI creation. On my understanding it
should only lookup for the specified addresses.
And to finish the tests, the second XS host remains equally during
tcp 0 48 10.7.0.102:22 192.168.172.1:58046
ESTABLISHED
udp 0 0 192.168.21.2:123 0.0.0.0:*
udp 0 0 192.168.20.2:123 0.0.0.0:*
As you said, perhaps propagating the initial connection through the
host will solve the problem without modifying the XenCenter
interface. A more sophisticated approach would be query the
network-list over XAPI, retrieve the PIFs associated to those
networks and them just check the IP addresses on the SR creation
request and match them with the equivalent PIFs.
For example here is the output of the xe network-param-list to
retrieve the associated PIF's and the xe pif-param-lst with the
uuid=4dc936d7-e806-c69a-a292-78e0ed2d5faa
uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa
name-label ( RW): iSCSI #0
PIF-uuids (SRO): 3baa8436-feca-cd04-3173-5002d9ffc66b;
362d63cb-647a-465f-6b81-1b59cd3b1f5d
MTU ( RW): 9000
bridge ( RO): xenbr2
other-config (MRW): automatic: true
default-locking-mode ( RW): unlocked
uuid=3baa8436-feca-cd04-3173-5002d9ffc66b
uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5c:64
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
host-name-label ( RO):xenserver2.local.iq.ufrj.br
<http://xenserver2.local.iq.ufrj.br/>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.20.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
uuid=362d63cb-647a-465f-6b81-1b59cd3b1f5d
uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5a:1c
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c
host-name-label ( RO):xenserver1.local.iq.ufrj.br
<http://xenserver1.local.iq.ufrj.br/>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.10.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
So as you can see, we have the IP addresses and the network mask.
Which is sufficient to an efficient and precise algorithm of SR creation.
I thinks this clarifies everything we discussed until now. A patch on
XenCenter appears to be really viable.
Many thanks,
Vinicius.
Yes, correct. The MD36xx looks like two separate controller
units, each acting as a separate device. For a standalone server
or some storage devices, that's right that you need a separate
subnet for each connection.
Because a single connection isn't seeing the broadcast from the
other connectors at the time of the initial connection from one
host, the pool is unaware of the others. The "repair" operation
causes this to be conducted from each host, and as I guessed,
then fixes the SR connectivity for the entire pool. I am
speculating that the request for a new connection of this type
via XenCenter could perhaps be solved by propagating that initial
connection process to the rest of the hosts in the pool and then
rescan the SR.
-=Tobias
Hi Tobias,
As for I know, the Dell MD36xx is a dual controller based
storage. So it's basically "two computers". Considering this with
the added knowledge that I've about TCP networking there's a
golden rule that says: only one IP address on a given subnet on a
single computer. So I can't use the same network on my case.
That's why I use point-to-point links too.
Since I've only one machine acting as a storage server (It's a
Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do
need four separate networks. FreeNAS is FreeBSD based, and BSD
enforces this rule by default. I can't have two IP addresses of
the same subnet on same machine, even if I manually configure it.
If this wasn't bad TCP network practice, I would be able to
configure the SR via XenCenter, since it will "find" the same
network on the others hosts without problem.
About the XenCenter, it really appears to be only a missing
function on the software. If I was a good programmer I would try
to implement this. But this is beyond my knowledge... :(
Thanks once again,
Vinícius.
No reason to necessarily use four separate subnets. You could
have 192.168.10.1. and 20.1 on one iSCSI controller and
192.168.10.2 and 2.2 on the other controller (on, for example, an
MD36xx that is standard). If however it is something like an
MD32xx which recommends four separate subets, then of course, use
four. One should follow the vendor's specifications.
I will respond in more detail later when I get a chance,
Vinícius, but after a quick read, I agree with your findings from
your other email. It would appear that the initial connection is
indeed evidently a missing function in XenCenter and perhaps all
that is needed to make this work "out of the box". As to the
wildcard discovery, it's generally preferred as it allows the
host to figure out the path on its own. The only reason I could
see not using a wildcard discovery would be if there were any
chance for overlaps and ambiguous connections.
Regards,
--Tobias
Hello Dave,
In total I have four paths. Two in each host. I've made a
drawning using ASCII characters, hope this can clarify the
+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+
192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30
192.168.10.2/30
192.168.11.2/30
192.168.20.2/30
192.168.21.2/30
That's why I'm using multipath. Because I do need MPIO to sum the
two network interfaces per host.
To exemplify a little more heres a screenshot of the network
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0
Cheers,
Vinícius.
Hi,
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it,
they recommend, and they even sell a solution using DELL hardware
with two machines and a shared storage running with 10GbE links.
Point-ot-point networks, dual controllers on the storage side and
two different network for iSCSI. They sell it at least here in
Brazil. What I’m trying to do is achieve the same topology with
XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a
better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different
markets. For example, I’ve used Fibre Channel only one time in my
life. And, believe it or not, it was a switchless configuration.
And it worked
 Wasn’t for VM workload but was in a GRID Cluster
with TORQUE/OpenPBS software. I do have the pictures of the
topology, but I need to search for it. At that time, I wasn’t the
guy who created the solution, but I understood that it was a huge
money saving, because the FC switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was
necessary in the CLI, but I was comfortable with this. The
following steps were necessary to reproduce
this:https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0
I’m really excited because it worked. So let’s me explain what
I’ve did.
First I started creating the SR via XenCenter, it failed as
expected. But to bypass I just created the SR via XenCenter with
the two Storage IP’s of the Pool Master. So the process went OK,
but it started broken, since the second XS host cannot see the IP
addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR
UUID created by XenCenter and the referenced PBDs. According to
what I know about XS, the PBD’s are the physical connect from
separate XS hosts to the shared storage. So it’s not related to
other hosts, it’s exclusively to the host machine that I’m using.
Understood that, the ideia is to destroy the created PBD on the
second XS host and recreate it with the proper settings and IP
addresses.
That’s when things got interesting. To create the PBD, I do need
the iSCSI working in the host, and consequently the multipath
connection. So I done the process in the terminal with the
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p
192.168.20.1:3260 --login
iscsiadm -m node --targetname
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p
192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
After all I was able to create the PBD with those new settings.
The tricky part was to pass the required “device-config” values
via the xe pbd-create command. I wasn’t aware of the method, but
after 4 tries I understood the process and typed the following
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f
host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|"
device-config:target=192.168.20.1,192.168.21.1
device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260
device-config:targetIQN=*
device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000
device-config:port=3260
I’d like to understand your configuration better— have you
effectively enabled 2 storage paths for the SR, knowing that each
host will only be able to use one path? Are your PBDs identical
or are there differences?
Thanks,
Dave
Basically everything necessary is in there. The IP’s, the name,
the multipath link, the LUN values, everything. After this, I
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
To prove that’s everything is working fine, I’ve created a new
VM, installed FreeBSD 10 on it, done a pkg install
xe-guest-utilities and requested a XenMotion operation. Oh boy,
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
Pics:https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0
I thinks this is sufficient for now, to prove that the concept
works and XenServer can create this kind of topology. It’s just
not available on the GUI. If XenCenter supports this kind of SR
on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was
really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to
try to squeeze something out of a technology that is not built-in
or future-safe. Would you want to try to connect a fiber-channel
storage device without a fiber channel switch (a very similar
situation)? There are other alternatives, for example, to
connect iSCSi storage to another, XenServer-independent server
and then serve storage from it via NFS or use NFS in the first
place and not make use at all of iSCSI technology. NFS has come a
long way and in many cases, can be equivalent in I/O performance
to iSCSI.
The intercommunications and ability to share SRs within a pool
are a significant consideration and cannot be ignored as part of
the storage support protocols within XenServer. XenServer
provides a number of options for storage connectivity to cover a
wide range of needs, preferences and costs. Often, trying to save
money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism
altogether (obviously, not a supported configuration) and connect
at least Linux VMs directly to iSCSI storage using open iSCSI,
but I have not explored this without still involving a network
switch. As you indicated, it's also not clear if XenServer will
remain 100% open iSCSI compatible or not in the future. It also
takes aways some of the things an SR can offer, like VM cloning,
storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors
in in improving performance. Networks switches are much more
capable and much faster than any storage device or host in
routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of
using a iSCSI SR without a Switch. The main goal is to reduce the
TCO of the solution, get some extra performance removing the
overhead that a managed switch puts on the network and reduce the
complexity of the network.
At a first glance I thought that I will only achieve those kind
of setup with a distributed switch, so DVSC should be an option.
But as Tim said it’s only a interface to control some policies.
I’ve looked for help on ServerFault and FreeNAS Forums, since I’m
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/
It appears that XenServer cannot support switchless iSCSI
storages without a switch, because XS needs the same network
block on all the XS hosts to create a shared storage between the
hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate
PBD’s. Perhaps one way to solve this issue is ignoring XenCenter
on SR creation and try to create an SR with all the PBD’s from
the XS hosts. But I was unable to test this setting due to lack
of my knowledge when trying to create and manipulate the SR via
the CLI.
Anyway, I’m opening the discussion here to get some feedback of
the developers, to see if this method of building the iSCSI
network is viable, since as Tim said, XS is not 100% open-iscsi
upstream and if it could be integrated on next versions of
XenServer via the XenCenter GUI. I think that’s a very common
architecture and VMware is doing this on small pools! So let’s do
it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Dave Scott
2015-02-18 18:50:39 UTC
Permalink
Regarding Point 3) I believe what is meant is that you should test having more than one SR that you are connecting to using the same IP addresses for the target array. You know it works fine to create one SR, so can you add additional ones the same way and have them all live together in harmony? :-)
As to the sr-create command, the host-uuid is an optional parameter and hence does not need to be named for the pool master or any other. The only required parameters are the name-label and the SR type. Have you tried that to see if the behavior is any different compared to when you do supply a host name?
I believe the underlying Xapi data model can express what you want, but there are 1 or 2 places where Xapi will helpfully create PBDs for you by cloning the master, which you just need to be aware of.

For example you should be able to use something like

$ xe sr-create .. host-uuid=<any host> shared=false

Setting “shared=false” initially should avoid Xapi ‘helpfully’ pre-creating identical PBDs for every host with the same configuration[1]. You should end up with an SR attached to a single host.

If you then set the SR to shared=true you should be able to create and plug the exact PBDs you want:

$ xe sr-param-set uuid= shared=true

$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...
$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...

[ as an aside, it’s obviously not great that the PBDs you get depend on exactly when you declare the SR is shared. This has happened because the original design had all PBDs being distinct, and the notion of ‘copying the master’ was added relatively late ]

Whenever a new host is added to the pool, Xapi will clone the master’s PBD for you— you would have to destroy it and re-create it. I suppose cloning the master’s configuration is as good a guess as any in this situation :-)

Dave

[1] https://github.com/xapi-project/xen-api/blob/b6f28c52fd9726553968b410e6b8cced5401f385/ocaml/xapi/xapi_sr.ml#L209
-=Tobias
Post by Vinícius Ferrão
Hello guys,
Answering about all the questions and issues. For those who aren't understanding the network topology, I've made a drawing of the solution. The image is here: https://www.dropbox.com/s/6zwerk6igcyqou3/UFRJ-IQ-VirtualPool.jpg?dl=0
192.168.10.1
192.168.11.1
192.168.20.1
192.168.21.1
The convention to this numbering scheme was: 192.168.{XenServerNumber}{InterfaceNumber}.{Storage(1)/Host(2)}.
192.168.10.2 (XenServer #1 - Interface #0)
192.168.11.2 (XenServer #1 - Interface #1)
192.168.20.2 (XenServer #2 - Interface #0)
192.168.21.2 (XenServer #2 - Interface #1)
Hope this completely clarifies the architecture/topology of the iSCSI network.
Restarting both hosts on the same time. [OK]
Restarting the second host. [OK]
Restarting the pool master. [OK]
2. HA is working too. It fences only the damaged host. I even tried a VM Failover. HA happened without any problem and the lost VM has been restarted on other server. To do the HA test, I've power cycled the host, simulated a hardware crash scenario. A cool picture: https://www.dropbox.com/s/g7h7rljk8vh97rg/Screenshot%202015-02-18%2015.45.41.png?dl=0
3. I haven't understood this test. You are asking to create a new LUN and add a new Shared Repository? It's not clear what you're saying with "independent SR".
4. Storage XenMotion works too. I've moved a running VM from local storage (on the Host #2) to the iSCSI pool without any problem. The reverse process works too.
5. I will test this tomorrow, when I will got physical access to the servers (carnival holiday here in BR). But about this specific test, I can't see any problem with plain Xen Motion, since the data is copied over the management network and not the iSCSI network. But I can see a good point when dealing with Storage XenMotion.
Finally about the changes on XenCenter or XenServer/XAPI itself. One thing to prove this point is to create the SR via the CLI. If I would be able to do this, the problem should definitely be on XenCenter. As I said before, I've created a broken Software iSCSI SR via XenCenter and them mapped the iSCSI interfaces, enabled multipath and created the PBD over the CLI.
The iSCSI and Multipath should be done in the physical host. But the PBD creation can be done without any problem on the Pool Master.
device-config: name-label= shared= type=
In this case sm-config and device-config are the problematic ones. Assuming that the host-uuid is always the pool master.
Thanks in advance,
Vinícius.
I’ll add in a few more tests….
1. Host restart. Does the normal PBD plug mechanism work with such a configuration
2. HA. While HA with two hosts isn’t a supported configuration, in this model it might work properly. If you remove the network cables for a single host, does it fence, or do both hosts fence?
3. Multiple SRs. If you have multiple LUNs on the FreeNAS exported and then used as independent SRs, does everything work as it should?
4. Storage XenMotion. With a vdi on one SR presented with the FreeNAS, copy it to any another storage option
5. Motion during path failure. With one path down on a given host, does XenMotion and Storage XenMotion still function correctly?
For those who are questioning the value of such a configuration, this is in essence a “cluster in a box” solution. These configurations were very popular 10-15 years ago as highly available compute infrastructure was quite expensive back then. Today we can accomplish the same things we did back then using virtualization, but “cluster in a box” can always be valuable for those with limited IT skills or for smaller organizations wanting least cost with fewest points of failure.
-tim
Sent: Monday, February 16, 2015 7:24 PM
To: Tobias Kreidl
Subject: Re: [xs-devel] Switchless shared iSCSI Network with /30 links
Tobias,
I've done more tests about the SR creation over XenCenter.
All the process definitely occurs only on the pool master. Even if the specified networks are not part of the networks of the host. The connection is tried on the default route.
192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1
The target IQN is discovered without any problem, since the iSCSI Storage reports the LUNs on any interface.
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 192.168.10.2:55870 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52654 192.168.11.1:3260 TIME_WAIT
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.11.2:52656 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
******************
tcp 0 1 10.7.0.101:56481 192.168.21.1:3260 SYN_SENT
******************
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
******************
tcp 0 1 10.7.0.101:52787 192.168.30.1:3260 SYN_SENT
******************
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
If you're wondering the 192.168.30.1 is reported because in the future we will add another XS host to the pool. The point here is that XenCenter should not lookup for this network since it isn't specified during the Software iSCSI creation. On my understanding it should only lookup for the specified addresses.
tcp 0 48 10.7.0.102:22 192.168.172.1:58046 ESTABLISHED
udp 0 0 192.168.21.2:123 0.0.0.0:*
udp 0 0 192.168.20.2:123 0.0.0.0:*
As you said, perhaps propagating the initial connection through the host will solve the problem without modifying the XenCenter interface. A more sophisticated approach would be query the network-list over XAPI, retrieve the PIFs associated to those networks and them just check the IP addresses on the SR creation request and match them with the equivalent PIFs.
uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa
name-label ( RW): iSCSI #0
PIF-uuids (SRO): 3baa8436-feca-cd04-3173-5002d9ffc66b; 362d63cb-647a-465f-6b81-1b59cd3b1f5d
MTU ( RW): 9000
bridge ( RO): xenbr2
other-config (MRW): automatic: true
default-locking-mode ( RW): unlocked
uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5c:64
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
host-name-label ( RO): xenserver2.local.iq.ufrj.br
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.20.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5a:1c
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c
host-name-label ( RO): xenserver1.local.iq.ufrj.br
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.10.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
So as you can see, we have the IP addresses and the network mask. Which is sufficient to an efficient and precise algorithm of SR creation.
I thinks this clarifies everything we discussed until now. A patch on XenCenter appears to be really viable.
Many thanks,
Vinicius.
Yes, correct. The MD36xx looks like two separate controller units, each acting as a separate device. For a standalone server or some storage devices, that's right that you need a separate subnet for each connection.
Because a single connection isn't seeing the broadcast from the other connectors at the time of the initial connection from one host, the pool is unaware of the others. The "repair" operation causes this to be conducted from each host, and as I guessed, then fixes the SR connectivity for the entire pool. I am speculating that the request for a new connection of this type via XenCenter could perhaps be solved by propagating that initial connection process to the rest of the hosts in the pool and then rescan the SR.
-=Tobias
Hi Tobias,
As for I know, the Dell MD36xx is a dual controller based storage. So it's basically "two computers". Considering this with the added knowledge that I've about TCP networking there's a golden rule that says: only one IP address on a given subnet on a single computer. So I can't use the same network on my case. That's why I use point-to-point links too.
Since I've only one machine acting as a storage server (It's a Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do need four separate networks. FreeNAS is FreeBSD based, and BSD enforces this rule by default. I can't have two IP addresses of the same subnet on same machine, even if I manually configure it.
If this wasn't bad TCP network practice, I would be able to configure the SR via XenCenter, since it will "find" the same network on the others hosts without problem.
About the XenCenter, it really appears to be only a missing function on the software. If I was a good programmer I would try to implement this. But this is beyond my knowledge... :(
Thanks once again,
Vinícius.
No reason to necessarily use four separate subnets. You could have 192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and 2.2 on the other controller (on, for example, an MD36xx that is standard). If however it is something like an MD32xx which recommends four separate subets, then of course, use four. One should follow the vendor's specifications.
I will respond in more detail later when I get a chance, Vinícius, but after a quick read, I agree with your findings from your other email. It would appear that the initial connection is indeed evidently a missing function in XenCenter and perhaps all that is needed to make this work "out of the box". As to the wildcard discovery, it's generally preferred as it allows the host to figure out the path on its own. The only reason I could see not using a wildcard discovery would be if there were any chance for overlaps and ambiguous connections.
Regards,
--Tobias
Hello Dave,
+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+
192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30
192.168.10.2/30
192.168.11.2/30
192.168.20.2/30
192.168.21.2/30
That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0
Cheers,
Vinícius.
Hi,
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked… Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0
I’m really excited because it worked. So let’s me explain what I’ve did.
First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I’d like to understand your configuration better— have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?
Thanks,
Dave
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0
I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/
It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Tobias Kreidl
2015-02-18 19:30:04 UTC
Permalink
Hi, Dave:
Thank you for your feedback.

Is not the whole idea of a shared SR such that a unique PBD is created
for each host that plugs in the same SR? When you add a new host to a
XenServer with "conventional" SRs, that's what it does. For example, the
command "xe sr-list uuid=(UUID-of-SR) params=PBDs" lists four unique
PBDs for my pool of four hosts, as it should. So I am confused about
the need to destroy the master PBD when a new host is added.

Back to the iSCSI SR creation. The fact that an "SR repair" operation
fixes everything is a sign that there is a good mechanism built in that
can handle this. Where it starts looking interesting is around line 178
where the SR is initially created as it states with "creates PDB [sic]
record for current host" and then as you pointed out, on line 209 is
where it should loop through all hosts, which for some reason in this
particular case, it doesn't seem to be doing when done via XenCenter.
It's not clear at least to me where it makes available the iSCSI target
address(es) available to pick up on all this.

What would be really helpful (to me, at least) I think would be a list
of the sequence of events or entry points that an "SR repair" option
from XenCenter initiates.

-=Tobias
Post by Dave Scott
Regarding Point 3) I believe what is meant is that you should test having more than one SR that you are connecting to using the same IP addresses for the target array. You know it works fine to create one SR, so can you add additional ones the same way and have them all live together in harmony? :-)
As to the sr-create command, the host-uuid is an optional parameter and hence does not need to be named for the pool master or any other. The only required parameters are the name-label and the SR type. Have you tried that to see if the behavior is any different compared to when you do supply a host name?
I believe the underlying Xapi data model can express what you want, but there are 1 or 2 places where Xapi will helpfully create PBDs for you by cloning the master, which you just need to be aware of.
For example you should be able to use something like
$ xe sr-create .. host-uuid=<any host> shared=false
Setting “shared=false” initially should avoid Xapi ‘helpfully’ pre-creating identical PBDs for every host with the same configuration[1]. You should end up with an SR attached to a single host.
$ xe sr-param-set uuid= shared=true
$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...
$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...
[ as an aside, it’s obviously not great that the PBDs you get depend on exactly when you declare the SR is shared. This has happened because the original design had all PBDs being distinct, and the notion of ‘copying the master’ was added relatively late ]
Whenever a new host is added to the pool, Xapi will clone the master’s PBD for you— you would have to destroy it and re-create it. I suppose cloning the master’s configuration is as good a guess as any in this situation :-)
Dave
[1] https://github.com/xapi-project/xen-api/blob/b6f28c52fd9726553968b410e6b8cced5401f385/ocaml/xapi/xapi_sr.ml#L209
-=Tobias
Post by Vinícius Ferrão
Hello guys,
Answering about all the questions and issues. For those who aren't understanding the network topology, I've made a drawing of the solution. The image is here: https://www.dropbox.com/s/6zwerk6igcyqou3/UFRJ-IQ-VirtualPool.jpg?dl=0
192.168.10.1
192.168.11.1
192.168.20.1
192.168.21.1
The convention to this numbering scheme was: 192.168.{XenServerNumber}{InterfaceNumber}.{Storage(1)/Host(2)}.
192.168.10.2 (XenServer #1 - Interface #0)
192.168.11.2 (XenServer #1 - Interface #1)
192.168.20.2 (XenServer #2 - Interface #0)
192.168.21.2 (XenServer #2 - Interface #1)
Hope this completely clarifies the architecture/topology of the iSCSI network.
Restarting both hosts on the same time. [OK]
Restarting the second host. [OK]
Restarting the pool master. [OK]
2. HA is working too. It fences only the damaged host. I even tried a VM Failover. HA happened without any problem and the lost VM has been restarted on other server. To do the HA test, I've power cycled the host, simulated a hardware crash scenario. A cool picture: https://www.dropbox.com/s/g7h7rljk8vh97rg/Screenshot%202015-02-18%2015.45.41.png?dl=0
3. I haven't understood this test. You are asking to create a new LUN and add a new Shared Repository? It's not clear what you're saying with "independent SR".
4. Storage XenMotion works too. I've moved a running VM from local storage (on the Host #2) to the iSCSI pool without any problem. The reverse process works too.
5. I will test this tomorrow, when I will got physical access to the servers (carnival holiday here in BR). But about this specific test, I can't see any problem with plain Xen Motion, since the data is copied over the management network and not the iSCSI network. But I can see a good point when dealing with Storage XenMotion.
Finally about the changes on XenCenter or XenServer/XAPI itself. One thing to prove this point is to create the SR via the CLI. If I would be able to do this, the problem should definitely be on XenCenter. As I said before, I've created a broken Software iSCSI SR via XenCenter and them mapped the iSCSI interfaces, enabled multipath and created the PBD over the CLI.
The iSCSI and Multipath should be done in the physical host. But the PBD creation can be done without any problem on the Pool Master.
device-config: name-label= shared= type=
In this case sm-config and device-config are the problematic ones. Assuming that the host-uuid is always the pool master.
Thanks in advance,
Vinícius.
Post by Tim Mackey
I’ll add in a few more tests
.
1. Host restart. Does the normal PBD plug mechanism work with such a configuration
2. HA. While HA with two hosts isn’t a supported configuration, in this model it might work properly. If you remove the network cables for a single host, does it fence, or do both hosts fence?
3. Multiple SRs. If you have multiple LUNs on the FreeNAS exported and then used as independent SRs, does everything work as it should?
4. Storage XenMotion. With a vdi on one SR presented with the FreeNAS, copy it to any another storage option
5. Motion during path failure. With one path down on a given host, does XenMotion and Storage XenMotion still function correctly?
For those who are questioning the value of such a configuration, this is in essence a “cluster in a box” solution. These configurations were very popular 10-15 years ago as highly available compute infrastructure was quite expensive back then. Today we can accomplish the same things we did back then using virtualization, but “cluster in a box” can always be valuable for those with limited IT skills or for smaller organizations wanting least cost with fewest points of failure.
-tim
Sent: Monday, February 16, 2015 7:24 PM
To: Tobias Kreidl
Subject: Re: [xs-devel] Switchless shared iSCSI Network with /30 links
Tobias,
I've done more tests about the SR creation over XenCenter.
All the process definitely occurs only on the pool master. Even if the specified networks are not part of the networks of the host. The connection is tried on the default route.
192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1
The target IQN is discovered without any problem, since the iSCSI Storage reports the LUNs on any interface.
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 192.168.10.2:55870 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52654 192.168.11.1:3260 TIME_WAIT
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.11.2:52656 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
******************
tcp 0 1 10.7.0.101:56481 192.168.21.1:3260 SYN_SENT
******************
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
******************
tcp 0 1 10.7.0.101:52787 192.168.30.1:3260 SYN_SENT
******************
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
If you're wondering the 192.168.30.1 is reported because in the future we will add another XS host to the pool. The point here is that XenCenter should not lookup for this network since it isn't specified during the Software iSCSI creation. On my understanding it should only lookup for the specified addresses.
tcp 0 48 10.7.0.102:22 192.168.172.1:58046 ESTABLISHED
udp 0 0 192.168.21.2:123 0.0.0.0:*
udp 0 0 192.168.20.2:123 0.0.0.0:*
As you said, perhaps propagating the initial connection through the host will solve the problem without modifying the XenCenter interface. A more sophisticated approach would be query the network-list over XAPI, retrieve the PIFs associated to those networks and them just check the IP addresses on the SR creation request and match them with the equivalent PIFs.
uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa
name-label ( RW): iSCSI #0
PIF-uuids (SRO): 3baa8436-feca-cd04-3173-5002d9ffc66b; 362d63cb-647a-465f-6b81-1b59cd3b1f5d
MTU ( RW): 9000
bridge ( RO): xenbr2
other-config (MRW): automatic: true
default-locking-mode ( RW): unlocked
uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5c:64
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
host-name-label ( RO): xenserver2.local.iq.ufrj.br
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.20.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5a:1c
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c
host-name-label ( RO): xenserver1.local.iq.ufrj.br
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.10.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
So as you can see, we have the IP addresses and the network mask. Which is sufficient to an efficient and precise algorithm of SR creation.
I thinks this clarifies everything we discussed until now. A patch on XenCenter appears to be really viable.
Many thanks,
Vinicius.
Yes, correct. The MD36xx looks like two separate controller units, each acting as a separate device. For a standalone server or some storage devices, that's right that you need a separate subnet for each connection.
Because a single connection isn't seeing the broadcast from the other connectors at the time of the initial connection from one host, the pool is unaware of the others. The "repair" operation causes this to be conducted from each host, and as I guessed, then fixes the SR connectivity for the entire pool. I am speculating that the request for a new connection of this type via XenCenter could perhaps be solved by propagating that initial connection process to the rest of the hosts in the pool and then rescan the SR.
-=Tobias
Hi Tobias,
As for I know, the Dell MD36xx is a dual controller based storage. So it's basically "two computers". Considering this with the added knowledge that I've about TCP networking there's a golden rule that says: only one IP address on a given subnet on a single computer. So I can't use the same network on my case. That's why I use point-to-point links too.
Since I've only one machine acting as a storage server (It's a Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do need four separate networks. FreeNAS is FreeBSD based, and BSD enforces this rule by default. I can't have two IP addresses of the same subnet on same machine, even if I manually configure it.
If this wasn't bad TCP network practice, I would be able to configure the SR via XenCenter, since it will "find" the same network on the others hosts without problem.
About the XenCenter, it really appears to be only a missing function on the software. If I was a good programmer I would try to implement this. But this is beyond my knowledge... :(
Thanks once again,
Vinícius.
No reason to necessarily use four separate subnets. You could have 192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and 2.2 on the other controller (on, for example, an MD36xx that is standard). If however it is something like an MD32xx which recommends four separate subets, then of course, use four. One should follow the vendor's specifications.
I will respond in more detail later when I get a chance, Vinícius, but after a quick read, I agree with your findings from your other email. It would appear that the initial connection is indeed evidently a missing function in XenCenter and perhaps all that is needed to make this work "out of the box". As to the wildcard discovery, it's generally preferred as it allows the host to figure out the path on its own. The only reason I could see not using a wildcard discovery would be if there were any chance for overlaps and ambiguous connections.
Regards,
--Tobias
Hello Dave,
+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+
192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30
192.168.10.2/30
192.168.11.2/30
192.168.20.2/30
192.168.21.2/30
That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0
Cheers,
Vinícius.
Hi,
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked
 Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0
I’m really excited because it worked. So let’s me explain what I’ve did.
First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I’d like to understand your configuration better— have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?
Thanks,
Dave
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0
I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/
It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Stephen Turner
2015-02-19 09:32:18 UTC
Permalink
The code for that is at https://github.com/xenserver/xenadmin/blob/master/XenModel/Actions/SR/SrRepairAction.cs, but basically it’s: on each host, do PBD.create and/or PBD.plug as necessary.
--
Stephen Turner


From: xs-devel-***@lists.xenserver.org [mailto:xs-devel-***@lists.xenserver.org] On Behalf Of Tobias Kreidl
Sent: 18 February 2015 19:30
To: xs-***@lists.xenserver.org
Subject: Re: [xs-devel] Switchless shared iSCSI Network with /30 links

Hi, Dave:
Thank you for your feedback.

Is not the whole idea of a shared SR such that a unique PBD is created for each host that plugs in the same SR? When you add a new host to a XenServer with "conventional" SRs, that's what it does. For example, the command "xe sr-list uuid=(UUID-of-SR) params=PBDs" lists four unique PBDs for my pool of four hosts, as it should. So I am confused about the need to destroy the master PBD when a new host is added.

Back to the iSCSI SR creation. The fact that an "SR repair" operation fixes everything is a sign that there is a good mechanism built in that can handle this. Where it starts looking interesting is around line 178 where the SR is initially created as it states with "creates PDB [sic]
record for current host" and then as you pointed out, on line 209 is where it should loop through all hosts, which for some reason in this particular case, it doesn't seem to be doing when done via XenCenter. It's not clear at least to me where it makes available the iSCSI target address(es) available to pick up on all this.

What would be really helpful (to me, at least) I think would be a list of the sequence of events or entry points that an "SR repair" option from XenCenter initiates.

-=Tobias

On 2/18/2015 11:50 AM, Dave Scott wrote:



On 18 Feb 2015, at 18:03, Tobias Kreidl <***@nau.edu><mailto:***@nau.edu> wrote:



Vinícius:



Regarding Point 3) I believe what is meant is that you should test having more than one SR that you are connecting to using the same IP addresses for the target array. You know it works fine to create one SR, so can you add additional ones the same way and have them all live together in harmony? :-)



As to the sr-create command, the host-uuid is an optional parameter and hence does not need to be named for the pool master or any other. The only required parameters are the name-label and the SR type. Have you tried that to see if the behavior is any different compared to when you do supply a host name?



I believe the underlying Xapi data model can express what you want, but there are 1 or 2 places where Xapi will helpfully create PBDs for you by cloning the master, which you just need to be aware of.



For example you should be able to use something like



$ xe sr-create .. host-uuid=<any host> shared=false



Setting “shared=false” initially should avoid Xapi ‘helpfully’ pre-creating identical PBDs for every host with the same configuration[1]. You should end up with an SR attached to a single host.



If you then set the SR to shared=true you should be able to create and plug the exact PBDs you want:



$ xe sr-param-set uuid= shared=true



$ xe pbd-create sr-uuid= host-uuid=

$ xe pbd-plug ...

$ xe pbd-create sr-uuid= host-uuid=

$ xe pbd-plug ...



[ as an aside, it’s obviously not great that the PBDs you get depend on exactly when you declare the SR is shared. This has happened because the original design had all PBDs being distinct, and the notion of ‘copying the master’ was added relatively late ]



Whenever a new host is added to the pool, Xapi will clone the master’s PBD for you— you would have to destroy it and re-create it. I suppose cloning the master’s configuration is as good a guess as any in this situation :-)



Dave



[1] https://github.com/xapi-project/xen-api/blob/b6f28c52fd9726553968b410e6b8cced5401f385/ocaml/xapi/xapi_sr.ml#L209





-=Tobias



On 2/18/2015 10:47 AM, Vinícius Ferrão wrote:

Hello guys,



Answering about all the questions and issues. For those who aren't understanding the network topology, I've made a drawing of the solution. The image is here: https://www.dropbox.com/s/6zwerk6igcyqou3/UFRJ-IQ-VirtualPool.jpg?dl=0



In the drawing you can see the iSCSI Network made with point-to-point links. The IP's on the storage box are:

192.168.10.1

192.168.11.1

192.168.20.1

192.168.21.1



The convention to this numbering scheme was: 192.168.{XenServerNumber}{InterfaceNumber}.{Storage(1)/Host(2)}.



So on the host #1 I've two addresses and more two on host #2:

192.168.10.2 (XenServer #1 - Interface #0)

192.168.11.2 (XenServer #1 - Interface #1)

192.168.20.2 (XenServer #2 - Interface #0)

192.168.21.2 (XenServer #2 - Interface #1)



Hope this completely clarifies the architecture/topology of the iSCSI network.



Talking about the tests raised by Tim:



1. Already done this test. Tried three different tests:





Restarting both hosts on the same time. [OK]





Restarting the second host. [OK]





Restarting the pool master. [OK]



2. HA is working too. It fences only the damaged host. I even tried a VM Failover. HA happened without any problem and the lost VM has been restarted on other server. To do the HA test, I've power cycled the host, simulated a hardware crash scenario. A cool picture: https://www.dropbox.com/s/g7h7rljk8vh97rg/Screenshot%202015-02-18%2015.45.41.png?dl=0



3. I haven't understood this test. You are asking to create a new LUN and add a new Shared Repository? It's not clear what you're saying with "independent SR".



4. Storage XenMotion works too. I've moved a running VM from local storage (on the Host #2) to the iSCSI pool without any problem. The reverse process works too.



5. I will test this tomorrow, when I will got physical access to the servers (carnival holiday here in BR). But about this specific test, I can't see any problem with plain Xen Motion, since the data is copied over the management network and not the iSCSI network. But I can see a good point when dealing with Storage XenMotion.



Finally about the changes on XenCenter or XenServer/XAPI itself. One thing to prove this point is to create the SR via the CLI. If I would be able to do this, the problem should definitely be on XenCenter. As I said before, I've created a broken Software iSCSI SR via XenCenter and them mapped the iSCSI interfaces, enabled multipath and created the PBD over the CLI.



The iSCSI and Multipath should be done in the physical host. But the PBD creation can be done without any problem on the Pool Master.



To test this scenario I do require some help crafting the xe sr-create command, because I don't know exactly how this command is formed by XenCenter, and it requires some doubtful inputs:



[***@xenserver1 ~]# xe sr-create

content-type= host-uuid= physical-size= sm-config:

device-config: name-label= shared= type=



In this case sm-config and device-config are the problematic ones. Assuming that the host-uuid is always the pool master.



Thanks in advance,

Vinícius.



On Feb 18, 2015, at 2:36 PM, Tim Mackey <***@citrix.com><mailto:***@citrix.com> wrote:



I’ll add in a few more tests
.



1. Host restart. Does the normal PBD plug mechanism work with such a configuration

2. HA. While HA with two hosts isn’t a supported configuration, in this model it might work properly. If you remove the network cables for a single host, does it fence, or do both hosts fence?

3. Multiple SRs. If you have multiple LUNs on the FreeNAS exported and then used as independent SRs, does everything work as it should?

4. Storage XenMotion. With a vdi on one SR presented with the FreeNAS, copy it to any another storage option

5. Motion during path failure. With one path down on a given host, does XenMotion and Storage XenMotion still function correctly?



For those who are questioning the value of such a configuration, this is in essence a “cluster in a box” solution. These configurations were very popular 10-15 years ago as highly available compute infrastructure was quite expensive back then. Today we can accomplish the same things we did back then using virtualization, but “cluster in a box” can always be valuable for those with limited IT skills or for smaller organizations wanting least cost with fewest points of failure.



-tim



From: xs-devel-***@lists.xenserver.org<mailto:xs-devel-***@lists.xenserver.org> [mailto:xs-devel-***@lists.xenserver.org] On Behalf Of Vinícius Ferrão

Sent: Monday, February 16, 2015 7:24 PM

To: Tobias Kreidl

Cc: xs-***@lists.xenserver.org<mailto:xs-***@lists.xenserver.org>

Subject: Re: [xs-devel] Switchless shared iSCSI Network with /30 links



Tobias,



I've done more tests about the SR creation over XenCenter.



All the process definitely occurs only on the pool master. Even if the specified networks are not part of the networks of the host. The connection is tried on the default route.



I requested a new Software iSCSI SR with the following address:

192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1



The target IQN is discovered without any problem, since the iSCSI Storage reports the LUNs on any interface.



But when trying to discover the LUN, the process only is done on the pool master. During the discovery process, I left a watch command on both XS to with "netstat -an | grep 192.168" and the results are:



On the pool master, there's a lot of things going on:

tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED

tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED

tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED

tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED

tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT

tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED

tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED

tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED

tcp 0 0 192.168.10.2:55870 192.168.10.1:3260 TIME_WAIT

tcp 0 0 192.168.11.2:52654 192.168.11.1:3260 TIME_WAIT

tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED

tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED

tcp 0 0 192.168.11.2:52656 192.168.11.1:3260 TIME_WAIT

tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT

udp 0 0 192.168.10.2:123 0.0.0.0:*

udp 0 0 192.168.11.2:123 0.0.0.0:*



During the discovery, look at the 192.168.21.1 address being searched through the default route:

tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED

tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT

tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED

tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED

tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED

tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT

tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT

tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT

tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT

tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT

tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED

tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED

tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT

tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED

******************

tcp 0 1 10.7.0.101:56481 192.168.21.1:3260 SYN_SENT

******************

tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED

tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED

tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT

udp 0 0 192.168.10.2:123 0.0.0.0:*

udp 0 0 192.168.11.2:123 0.0.0.0:*



Even addresses that does not exist on the pool (since the IQN discovery process reports them):

tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED

tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT

tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED

tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED

tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED

tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT

tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT

tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT

tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT

tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT

tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED

******************

tcp 0 1 10.7.0.101:52787 192.168.30.1:3260 SYN_SENT

******************

tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED

tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT

tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED

tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED

tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED

tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT

udp 0 0 192.168.10.2:123 0.0.0.0:*

udp 0 0 192.168.11.2:123 0.0.0.0:*



If you're wondering the 192.168.30.1 is reported because in the future we will add another XS host to the pool. The point here is that XenCenter should not lookup for this network since it isn't specified during the Software iSCSI creation. On my understanding it should only lookup for the specified addresses.



And to finish the tests, the second XS host remains equally during all the SR creation phase:

[***@xenserver2 ~]# netstat -an | grep -i 192.168

tcp 0 48 10.7.0.102:22 192.168.172.1:58046 ESTABLISHED

udp 0 0 192.168.21.2:123 0.0.0.0:*

udp 0 0 192.168.20.2:123 0.0.0.0:*



As you said, perhaps propagating the initial connection through the host will solve the problem without modifying the XenCenter interface. A more sophisticated approach would be query the network-list over XAPI, retrieve the PIFs associated to those networks and them just check the IP addresses on the SR creation request and match them with the equivalent PIFs.



For example here is the output of the xe network-param-list to retrieve the associated PIF's and the xe pif-param-lst with the UUID's of the retrieved PIFs:



[***@xenserver1 ~]# xe network-param-list uuid=4dc936d7-e806-c69a-a292-78e0ed2d5faa

uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa

name-label ( RW): iSCSI #0

name-description ( RW):

VIF-uuids (SRO):

PIF-uuids (SRO): 3baa8436-feca-cd04-3173-5002d9ffc66b; 362d63cb-647a-465f-6b81-1b59cd3b1f5d

MTU ( RW): 9000

bridge ( RO): xenbr2

other-config (MRW): automatic: true

blobs ( RO):

tags (SRW):

default-locking-mode ( RW): unlocked





[***@xenserver1 ~]# xe pif-param-list uuid=3baa8436-feca-cd04-3173-5002d9ffc66b

uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b

device ( RO): eth2

MAC ( RO): 40:f2:e9:f3:5c:64

physical ( RO): true

managed ( RO): true

currently-attached ( RO): true

MTU ( RO): 9000

VLAN ( RO): -1

bond-master-of ( RO):

bond-slave-of ( RO): <not in database>

tunnel-access-PIF-of ( RO):

tunnel-transport-PIF-of ( RO):

management ( RO): false

network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa

network-name-label ( RO): iSCSI #0

host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e

host-name-label ( RO): xenserver2.local.iq.ufrj.br

IP-configuration-mode ( RO): Static

IP ( RO): 192.168.20.2

netmask ( RO): 255.255.255.252

gateway ( RO):

IPv6-configuration-mode ( RO): None

IPv6 ( RO):

IPv6-gateway ( RO):

primary-address-type ( RO): IPv4

DNS ( RO):

properties (MRO): gro: on

io_read_kbs ( RO): 0.000

io_write_kbs ( RO): 0.000

carrier ( RO): true

vendor-id ( RO): 8086

vendor-name ( RO): Intel Corporation

device-id ( RO): 1521

device-name ( RO): I350 Gigabit Network Connection

speed ( RO): 1000 Mbit/s

duplex ( RO): full

disallow-unplug ( RW): true

pci-bus-path ( RO): 0000:06:00.2

other-config (MRW): management_purpose: iSCSI MPIO #0





[***@xenserver1 ~]# xe pif-param-list uuid=362d63cb-647a-465f-6b81-1b59cd3b1f5d

uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d

device ( RO): eth2

MAC ( RO): 40:f2:e9:f3:5a:1c

physical ( RO): true

managed ( RO): true

currently-attached ( RO): true

MTU ( RO): 9000

VLAN ( RO): -1

bond-master-of ( RO):

bond-slave-of ( RO): <not in database>

tunnel-access-PIF-of ( RO):

tunnel-transport-PIF-of ( RO):

management ( RO): false

network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa

network-name-label ( RO): iSCSI #0

host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c

host-name-label ( RO): xenserver1.local.iq.ufrj.br

IP-configuration-mode ( RO): Static

IP ( RO): 192.168.10.2

netmask ( RO): 255.255.255.252

gateway ( RO):

IPv6-configuration-mode ( RO): None

IPv6 ( RO):

IPv6-gateway ( RO):

primary-address-type ( RO): IPv4

DNS ( RO):

properties (MRO): gro: on

io_read_kbs ( RO): 0.000

io_write_kbs ( RO): 0.000

carrier ( RO): true

vendor-id ( RO): 8086

vendor-name ( RO): Intel Corporation

device-id ( RO): 1521

device-name ( RO): I350 Gigabit Network Connection

speed ( RO): 1000 Mbit/s

duplex ( RO): full

disallow-unplug ( RW): true

pci-bus-path ( RO): 0000:06:00.2

other-config (MRW): management_purpose: iSCSI MPIO #0



So as you can see, we have the IP addresses and the network mask. Which is sufficient to an efficient and precise algorithm of SR creation.



I thinks this clarifies everything we discussed until now. A patch on XenCenter appears to be really viable.



Many thanks,

Vinicius.



On Feb 16, 2015, at 9:02 PM, Tobias Kreidl <***@nau.edu><mailto:***@nau.edu> wrote:



Yes, correct. The MD36xx looks like two separate controller units, each acting as a separate device. For a standalone server or some storage devices, that's right that you need a separate subnet for each connection.



Because a single connection isn't seeing the broadcast from the other connectors at the time of the initial connection from one host, the pool is unaware of the others. The "repair" operation causes this to be conducted from each host, and as I guessed, then fixes the SR connectivity for the entire pool. I am speculating that the request for a new connection of this type via XenCenter could perhaps be solved by propagating that initial connection process to the rest of the hosts in the pool and then rescan the SR.



-=Tobias



On 2/16/2015 2:20 PM, Vinícius Ferrão wrote:



Hi Tobias,



As for I know, the Dell MD36xx is a dual controller based storage. So it's basically "two computers". Considering this with the added knowledge that I've about TCP networking there's a golden rule that says: only one IP address on a given subnet on a single computer. So I can't use the same network on my case. That's why I use point-to-point links too.



Since I've only one machine acting as a storage server (It's a Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do need four separate networks. FreeNAS is FreeBSD based, and BSD enforces this rule by default. I can't have two IP addresses of the same subnet on same machine, even if I manually configure it.



If this wasn't bad TCP network practice, I would be able to configure the SR via XenCenter, since it will "find" the same network on the others hosts without problem.



About the XenCenter, it really appears to be only a missing function on the software. If I was a good programmer I would try to implement this. But this is beyond my knowledge... :(



Thanks once again,

Vinícius.





On Feb 16, 2015, at 6:53 PM, Tobias Kreidl <***@nau.edu><mailto:***@nau.edu> wrote:



No reason to necessarily use four separate subnets. You could have 192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and 2.2 on the other controller (on, for example, an MD36xx that is standard). If however it is something like an MD32xx which recommends four separate subets, then of course, use four. One should follow the vendor's specifications.



I will respond in more detail later when I get a chance, Vinícius, but after a quick read, I agree with your findings from your other email. It would appear that the initial connection is indeed evidently a missing function in XenCenter and perhaps all that is needed to make this work "out of the box". As to the wildcard discovery, it's generally preferred as it allows the host to figure out the path on its own. The only reason I could see not using a wildcard discovery would be if there were any chance for overlaps and ambiguous connections.



Regards,

--Tobias





On 2/16/2015 1:38 PM, Vinícius Ferrão wrote:



Hello Dave,



In total I have four paths. Two in each host. I've made a drawning using ASCII characters, hope this can clarify the architecture:



+----------------+

| iSCSI Storage |

+--|1||2||3||4|--+

| | | |

| | | \--------------\ iSCSI Multipathed /30

| | \--------------\ | Network with two links

| | | |

+--|1||2|--------+ +--|1||2|--------+

| XenServer Host | | XenServer Host |

+----------------+ +----------------+



The Storage has four gigabit ethernet adapters with the following IP addresses:



192.168.10.1/30

192.168.11.1/30

192.168.20.1/30

192.168.21.1/30



On the first XenServer there are two interfaces with those IP's:



192.168.10.2/30

192.168.11.2/30



And on the second host the equivalent configuration:



192.168.20.2/30

192.168.21.2/30



That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.



To exemplify a little more heres a screenshot of the network configuration made on XenCenter:

https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0



Cheers,

Vinícius.





On Feb 15, 2015, at 12:22 PM, Dave Scott <***@citrix.com><mailto:***@citrix.com> wrote:



Hi,





On 15 Feb 2015, at 05:44, Vinícius Ferrão <***@ferrao.eti.br><mailto:***@ferrao.eti.br> wrote:



Hello Tobias,

Thanks for the reply, even to discourage me :)



I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.



Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.



Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked
 Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.



Leaving all those “political” points aside, let’s talk technically :)



I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0



I’m really excited because it worked. So let’s me explain what I’ve did.



First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.



Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.



That’s when things got interesting. To create the PBD, I do need the iSCSI working in the host, and consequently the multipath connection. So I done the process in the terminal with the following commands:



#Discovery of iSCSI Service

iscsiadm -m discovery -t sendtargets -p 192.168.20.1

iscsiadm -m discovery -t sendtargets -p 192.168.21.1



#Login in the iSCSI Targets

iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login

iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login



#Enable Multipath

multipath

multipath -ll



After all I was able to create the PBD with those new settings. The tricky part was to pass the required “device-config” values via the xe pbd-create command. I wasn’t aware of the method, but after 4 tries I understood the process and typed the following command:



#Create the appropriate PBD

xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260

I’d like to understand your configuration better— have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?



Thanks,

Dave





Basically everything necessary is in there. The IP’s, the name, the multipath link, the LUN values, everything. After this, I just plugged the PDB via the CLI:



#Plug the PBD

xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342



And BOOM! It worked. As you can see in the screenshot.



To prove that’s everything is working fine, I’ve created a new VM, installed FreeBSD 10 on it, done a pkg install xe-guest-utilities and requested a XenMotion operation. Oh boy, it worked:



Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248

PING 10.7.0.248 (10.7.0.248): 56 data bytes

64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms

64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms

64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms

64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms

64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms

64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms

64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms

64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms

64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms

64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms

64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms

64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms

64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms

64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms

64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms

64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms

64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms

64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms

64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms

64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms

Request timeout for icmp_seq 20

Request timeout for icmp_seq 21

Request timeout for icmp_seq 22

Request timeout for icmp_seq 23

Request timeout for icmp_seq 24

Request timeout for icmp_seq 25

64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms

64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms

64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms

64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms

64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms

64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms

64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms

64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms

64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms

64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms

64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms

^C

--- 10.7.0.248 ping statistics ---

37 packets transmitted, 31 packets received, 16.2% packet loss

round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms



Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0



I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.



What I ask now: a feedback, if possible.



Thanks in advance,

Vinícius.



PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.





On Feb 14, 2015, at 8:14 PM, Tobias Kreidl <***@nau.edu><mailto:***@nau.edu> wrote:



Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.



The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.



That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.



Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.



Regards,

-=Tobias



On 2/14/2015 2:26 PM, Vinícius Ferrão wrote:



Hello guys,



I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.



At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.



I’ve looked for help on ServerFault and FreeNAS Forums, since I’m running FreeNAS as iSCSI service. The discussion is going on those links:



http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch

https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/



It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.



If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.



Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)



Thanks in advance,

Vinicius.
Vinícius Ferrão
2015-02-20 22:20:06 UTC
Permalink
Dave,

I've added the "future" server to our pool. And happened exactly what you said. The configurations were cloned. But obviously it failed to clone the PDB's and the Network configuration for the iSCSI network.

IMHO there are two things that needs to be addressed:
1. The network configuration on added host should be "confirmed" to the user, and not done automatically, so the user can adapt is to specific needs.
2. After this the PDB creation should be done accordingly. If the network topology has changed by the user, so the PDBs must be created with this new topology.

In resume, I've three host at this moment on the pool. So I can do the HA tests once again, and I will report to the list the results. At this moment the only thing that failed is cloning the iSCSI network configurations and plugging the PDBs. Only the bounded NIC0+NIC1 was created on the added host.

Thanks in advance,
Vinícius.
Post by Dave Scott
Regarding Point 3) I believe what is meant is that you should test having more than one SR that you are connecting to using the same IP addresses for the target array. You know it works fine to create one SR, so can you add additional ones the same way and have them all live together in harmony? :-)
As to the sr-create command, the host-uuid is an optional parameter and hence does not need to be named for the pool master or any other. The only required parameters are the name-label and the SR type. Have you tried that to see if the behavior is any different compared to when you do supply a host name?
I believe the underlying Xapi data model can express what you want, but there are 1 or 2 places where Xapi will helpfully create PBDs for you by cloning the master, which you just need to be aware of.
For example you should be able to use something like
$ xe sr-create .. host-uuid=<any host> shared=false
Setting “shared=false” initially should avoid Xapi ‘helpfully’ pre-creating identical PBDs for every host with the same configuration[1]. You should end up with an SR attached to a single host.
$ xe sr-param-set uuid= shared=true
$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...
$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...
[ as an aside, it’s obviously not great that the PBDs you get depend on exactly when you declare the SR is shared. This has happened because the original design had all PBDs being distinct, and the notion of ‘copying the master’ was added relatively late ]
Whenever a new host is added to the pool, Xapi will clone the master’s PBD for you— you would have to destroy it and re-create it. I suppose cloning the master’s configuration is as good a guess as any in this situation :-)
Dave
[1] https://github.com/xapi-project/xen-api/blob/b6f28c52fd9726553968b410e6b8cced5401f385/ocaml/xapi/xapi_sr.ml#L209
-=Tobias
Post by Vinícius Ferrão
Hello guys,
Answering about all the questions and issues. For those who aren't understanding the network topology, I've made a drawing of the solution. The image is here: https://www.dropbox.com/s/6zwerk6igcyqou3/UFRJ-IQ-VirtualPool.jpg?dl=0
192.168.10.1
192.168.11.1
192.168.20.1
192.168.21.1
The convention to this numbering scheme was: 192.168.{XenServerNumber}{InterfaceNumber}.{Storage(1)/Host(2)}.
192.168.10.2 (XenServer #1 - Interface #0)
192.168.11.2 (XenServer #1 - Interface #1)
192.168.20.2 (XenServer #2 - Interface #0)
192.168.21.2 (XenServer #2 - Interface #1)
Hope this completely clarifies the architecture/topology of the iSCSI network.
Restarting both hosts on the same time. [OK]
Restarting the second host. [OK]
Restarting the pool master. [OK]
2. HA is working too. It fences only the damaged host. I even tried a VM Failover. HA happened without any problem and the lost VM has been restarted on other server. To do the HA test, I've power cycled the host, simulated a hardware crash scenario. A cool picture: https://www.dropbox.com/s/g7h7rljk8vh97rg/Screenshot%202015-02-18%2015.45.41.png?dl=0
3. I haven't understood this test. You are asking to create a new LUN and add a new Shared Repository? It's not clear what you're saying with "independent SR".
4. Storage XenMotion works too. I've moved a running VM from local storage (on the Host #2) to the iSCSI pool without any problem. The reverse process works too.
5. I will test this tomorrow, when I will got physical access to the servers (carnival holiday here in BR). But about this specific test, I can't see any problem with plain Xen Motion, since the data is copied over the management network and not the iSCSI network. But I can see a good point when dealing with Storage XenMotion.
Finally about the changes on XenCenter or XenServer/XAPI itself. One thing to prove this point is to create the SR via the CLI. If I would be able to do this, the problem should definitely be on XenCenter. As I said before, I've created a broken Software iSCSI SR via XenCenter and them mapped the iSCSI interfaces, enabled multipath and created the PBD over the CLI.
The iSCSI and Multipath should be done in the physical host. But the PBD creation can be done without any problem on the Pool Master.
device-config: name-label= shared= type=
In this case sm-config and device-config are the problematic ones. Assuming that the host-uuid is always the pool master.
Thanks in advance,
Vinícius.
I’ll add in a few more tests….
1. Host restart. Does the normal PBD plug mechanism work with such a configuration
2. HA. While HA with two hosts isn’t a supported configuration, in this model it might work properly. If you remove the network cables for a single host, does it fence, or do both hosts fence?
3. Multiple SRs. If you have multiple LUNs on the FreeNAS exported and then used as independent SRs, does everything work as it should?
4. Storage XenMotion. With a vdi on one SR presented with the FreeNAS, copy it to any another storage option
5. Motion during path failure. With one path down on a given host, does XenMotion and Storage XenMotion still function correctly?
For those who are questioning the value of such a configuration, this is in essence a “cluster in a box” solution. These configurations were very popular 10-15 years ago as highly available compute infrastructure was quite expensive back then. Today we can accomplish the same things we did back then using virtualization, but “cluster in a box” can always be valuable for those with limited IT skills or for smaller organizations wanting least cost with fewest points of failure.
-tim
Sent: Monday, February 16, 2015 7:24 PM
To: Tobias Kreidl
Subject: Re: [xs-devel] Switchless shared iSCSI Network with /30 links
Tobias,
I've done more tests about the SR creation over XenCenter.
All the process definitely occurs only on the pool master. Even if the specified networks are not part of the networks of the host. The connection is tried on the default route.
192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1
The target IQN is discovered without any problem, since the iSCSI Storage reports the LUNs on any interface.
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 192.168.10.2:55870 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52654 192.168.11.1:3260 TIME_WAIT
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.11.2:52656 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
******************
tcp 0 1 10.7.0.101:56481 192.168.21.1:3260 SYN_SENT
******************
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
******************
tcp 0 1 10.7.0.101:52787 192.168.30.1:3260 SYN_SENT
******************
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
If you're wondering the 192.168.30.1 is reported because in the future we will add another XS host to the pool. The point here is that XenCenter should not lookup for this network since it isn't specified during the Software iSCSI creation. On my understanding it should only lookup for the specified addresses.
tcp 0 48 10.7.0.102:22 192.168.172.1:58046 ESTABLISHED
udp 0 0 192.168.21.2:123 0.0.0.0:*
udp 0 0 192.168.20.2:123 0.0.0.0:*
As you said, perhaps propagating the initial connection through the host will solve the problem without modifying the XenCenter interface. A more sophisticated approach would be query the network-list over XAPI, retrieve the PIFs associated to those networks and them just check the IP addresses on the SR creation request and match them with the equivalent PIFs.
uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa
name-label ( RW): iSCSI #0
PIF-uuids (SRO): 3baa8436-feca-cd04-3173-5002d9ffc66b; 362d63cb-647a-465f-6b81-1b59cd3b1f5d
MTU ( RW): 9000
bridge ( RO): xenbr2
other-config (MRW): automatic: true
default-locking-mode ( RW): unlocked
uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5c:64
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
host-name-label ( RO): xenserver2.local.iq.ufrj.br
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.20.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5a:1c
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c
host-name-label ( RO): xenserver1.local.iq.ufrj.br
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.10.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
So as you can see, we have the IP addresses and the network mask. Which is sufficient to an efficient and precise algorithm of SR creation.
I thinks this clarifies everything we discussed until now. A patch on XenCenter appears to be really viable.
Many thanks,
Vinicius.
Yes, correct. The MD36xx looks like two separate controller units, each acting as a separate device. For a standalone server or some storage devices, that's right that you need a separate subnet for each connection.
Because a single connection isn't seeing the broadcast from the other connectors at the time of the initial connection from one host, the pool is unaware of the others. The "repair" operation causes this to be conducted from each host, and as I guessed, then fixes the SR connectivity for the entire pool. I am speculating that the request for a new connection of this type via XenCenter could perhaps be solved by propagating that initial connection process to the rest of the hosts in the pool and then rescan the SR.
-=Tobias
Hi Tobias,
As for I know, the Dell MD36xx is a dual controller based storage. So it's basically "two computers". Considering this with the added knowledge that I've about TCP networking there's a golden rule that says: only one IP address on a given subnet on a single computer. So I can't use the same network on my case. That's why I use point-to-point links too.
Since I've only one machine acting as a storage server (It's a Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do need four separate networks. FreeNAS is FreeBSD based, and BSD enforces this rule by default. I can't have two IP addresses of the same subnet on same machine, even if I manually configure it.
If this wasn't bad TCP network practice, I would be able to configure the SR via XenCenter, since it will "find" the same network on the others hosts without problem.
About the XenCenter, it really appears to be only a missing function on the software. If I was a good programmer I would try to implement this. But this is beyond my knowledge... :(
Thanks once again,
Vinícius.
No reason to necessarily use four separate subnets. You could have 192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and 2.2 on the other controller (on, for example, an MD36xx that is standard). If however it is something like an MD32xx which recommends four separate subets, then of course, use four. One should follow the vendor's specifications.
I will respond in more detail later when I get a chance, Vinícius, but after a quick read, I agree with your findings from your other email. It would appear that the initial connection is indeed evidently a missing function in XenCenter and perhaps all that is needed to make this work "out of the box". As to the wildcard discovery, it's generally preferred as it allows the host to figure out the path on its own. The only reason I could see not using a wildcard discovery would be if there were any chance for overlaps and ambiguous connections.
Regards,
--Tobias
Hello Dave,
+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+
192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30
192.168.10.2/30
192.168.11.2/30
192.168.20.2/30
192.168.21.2/30
That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0
Cheers,
Vinícius.
Hi,
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked… Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0
I’m really excited because it worked. So let’s me explain what I’ve did.
First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I’d like to understand your configuration better— have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?
Thanks,
Dave
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0
I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/
It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Tobias Kreidl
2015-02-20 22:37:30 UTC
Permalink
It may be possible to at least temporarily get this work by forcing the
SrRepairAction procedure to be run.
I didn't have time yet, but wanted to compare what it does in sequence
vs. what is done by default in the xapi_sr
program to see what is different. As far as XenCenter is concerned,
perhaps if multiple IP target addresses are entered in the dialog box
(as opposed to a single one or a wildcard entry), then a different means
could be triggered to ensure that each host does a proper PBD creation
and plug-in for that new SR.

-=Tobias
Post by Vinícius Ferrão
Dave,
I've added the "future" server to our pool. And happened exactly what you said. The configurations were cloned. But obviously it failed to clone the PDB's and the Network configuration for the iSCSI network.
1. The network configuration on added host should be "confirmed" to the user, and not done automatically, so the user can adapt is to specific needs.
2. After this the PDB creation should be done accordingly. If the network topology has changed by the user, so the PDBs must be created with this new topology.
In resume, I've three host at this moment on the pool. So I can do the HA tests once again, and I will report to the list the results. At this moment the only thing that failed is cloning the iSCSI network configurations and plugging the PDBs. Only the bounded NIC0+NIC1 was created on the added host.
Thanks in advance,
Vinícius.
Post by Dave Scott
Regarding Point 3) I believe what is meant is that you should test having more than one SR that you are connecting to using the same IP addresses for the target array. You know it works fine to create one SR, so can you add additional ones the same way and have them all live together in harmony? :-)
As to the sr-create command, the host-uuid is an optional parameter and hence does not need to be named for the pool master or any other. The only required parameters are the name-label and the SR type. Have you tried that to see if the behavior is any different compared to when you do supply a host name?
I believe the underlying Xapi data model can express what you want, but there are 1 or 2 places where Xapi will helpfully create PBDs for you by cloning the master, which you just need to be aware of.
For example you should be able to use something like
$ xe sr-create .. host-uuid=<any host> shared=false
Setting “shared=false” initially should avoid Xapi ‘helpfully’ pre-creating identical PBDs for every host with the same configuration[1]. You should end up with an SR attached to a single host.
$ xe sr-param-set uuid= shared=true
$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...
$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...
[ as an aside, it’s obviously not great that the PBDs you get depend on exactly when you declare the SR is shared. This has happened because the original design had all PBDs being distinct, and the notion of ‘copying the master’ was added relatively late ]
Whenever a new host is added to the pool, Xapi will clone the master’s PBD for you— you would have to destroy it and re-create it. I suppose cloning the master’s configuration is as good a guess as any in this situation :-)
Dave
[1] https://github.com/xapi-project/xen-api/blob/b6f28c52fd9726553968b410e6b8cced5401f385/ocaml/xapi/xapi_sr.ml#L209
-=Tobias
Post by Vinícius Ferrão
Hello guys,
Answering about all the questions and issues. For those who aren't understanding the network topology, I've made a drawing of the solution. The image is here: https://www.dropbox.com/s/6zwerk6igcyqou3/UFRJ-IQ-VirtualPool.jpg?dl=0
192.168.10.1
192.168.11.1
192.168.20.1
192.168.21.1
The convention to this numbering scheme was: 192.168.{XenServerNumber}{InterfaceNumber}.{Storage(1)/Host(2)}.
192.168.10.2 (XenServer #1 - Interface #0)
192.168.11.2 (XenServer #1 - Interface #1)
192.168.20.2 (XenServer #2 - Interface #0)
192.168.21.2 (XenServer #2 - Interface #1)
Hope this completely clarifies the architecture/topology of the iSCSI network.
Restarting both hosts on the same time. [OK]
Restarting the second host. [OK]
Restarting the pool master. [OK]
2. HA is working too. It fences only the damaged host. I even tried a VM Failover. HA happened without any problem and the lost VM has been restarted on other server. To do the HA test, I've power cycled the host, simulated a hardware crash scenario. A cool picture: https://www.dropbox.com/s/g7h7rljk8vh97rg/Screenshot%202015-02-18%2015.45.41.png?dl=0
3. I haven't understood this test. You are asking to create a new LUN and add a new Shared Repository? It's not clear what you're saying with "independent SR".
4. Storage XenMotion works too. I've moved a running VM from local storage (on the Host #2) to the iSCSI pool without any problem. The reverse process works too.
5. I will test this tomorrow, when I will got physical access to the servers (carnival holiday here in BR). But about this specific test, I can't see any problem with plain Xen Motion, since the data is copied over the management network and not the iSCSI network. But I can see a good point when dealing with Storage XenMotion.
Finally about the changes on XenCenter or XenServer/XAPI itself. One thing to prove this point is to create the SR via the CLI. If I would be able to do this, the problem should definitely be on XenCenter. As I said before, I've created a broken Software iSCSI SR via XenCenter and them mapped the iSCSI interfaces, enabled multipath and created the PBD over the CLI.
The iSCSI and Multipath should be done in the physical host. But the PBD creation can be done without any problem on the Pool Master.
device-config: name-label= shared= type=
In this case sm-config and device-config are the problematic ones. Assuming that the host-uuid is always the pool master.
Thanks in advance,
Vinícius.
I’ll add in a few more tests….
1. Host restart. Does the normal PBD plug mechanism work with such a configuration
2. HA. While HA with two hosts isn’t a supported configuration, in this model it might work properly. If you remove the network cables for a single host, does it fence, or do both hosts fence?
3. Multiple SRs. If you have multiple LUNs on the FreeNAS exported and then used as independent SRs, does everything work as it should?
4. Storage XenMotion. With a vdi on one SR presented with the FreeNAS, copy it to any another storage option
5. Motion during path failure. With one path down on a given host, does XenMotion and Storage XenMotion still function correctly?
For those who are questioning the value of such a configuration, this is in essence a “cluster in a box” solution. These configurations were very popular 10-15 years ago as highly available compute infrastructure was quite expensive back then. Today we can accomplish the same things we did back then using virtualization, but “cluster in a box” can always be valuable for those with limited IT skills or for smaller organizations wanting least cost with fewest points of failure.
-tim
Sent: Monday, February 16, 2015 7:24 PM
To: Tobias Kreidl
Subject: Re: [xs-devel] Switchless shared iSCSI Network with /30 links
Tobias,
I've done more tests about the SR creation over XenCenter.
All the process definitely occurs only on the pool master. Even if the specified networks are not part of the networks of the host. The connection is tried on the default route.
192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1
The target IQN is discovered without any problem, since the iSCSI Storage reports the LUNs on any interface.
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 192.168.10.2:55870 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52654 192.168.11.1:3260 TIME_WAIT
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.11.2:52656 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
******************
tcp 0 1 10.7.0.101:56481 192.168.21.1:3260 SYN_SENT
******************
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
******************
tcp 0 1 10.7.0.101:52787 192.168.30.1:3260 SYN_SENT
******************
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
If you're wondering the 192.168.30.1 is reported because in the future we will add another XS host to the pool. The point here is that XenCenter should not lookup for this network since it isn't specified during the Software iSCSI creation. On my understanding it should only lookup for the specified addresses.
tcp 0 48 10.7.0.102:22 192.168.172.1:58046 ESTABLISHED
udp 0 0 192.168.21.2:123 0.0.0.0:*
udp 0 0 192.168.20.2:123 0.0.0.0:*
As you said, perhaps propagating the initial connection through the host will solve the problem without modifying the XenCenter interface. A more sophisticated approach would be query the network-list over XAPI, retrieve the PIFs associated to those networks and them just check the IP addresses on the SR creation request and match them with the equivalent PIFs.
uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa
name-label ( RW): iSCSI #0
PIF-uuids (SRO): 3baa8436-feca-cd04-3173-5002d9ffc66b; 362d63cb-647a-465f-6b81-1b59cd3b1f5d
MTU ( RW): 9000
bridge ( RO): xenbr2
other-config (MRW): automatic: true
default-locking-mode ( RW): unlocked
uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5c:64
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
host-name-label ( RO): xenserver2.local.iq.ufrj.br
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.20.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5a:1c
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c
host-name-label ( RO): xenserver1.local.iq.ufrj.br
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.10.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
So as you can see, we have the IP addresses and the network mask. Which is sufficient to an efficient and precise algorithm of SR creation.
I thinks this clarifies everything we discussed until now. A patch on XenCenter appears to be really viable.
Many thanks,
Vinicius.
Yes, correct. The MD36xx looks like two separate controller units, each acting as a separate device. For a standalone server or some storage devices, that's right that you need a separate subnet for each connection.
Because a single connection isn't seeing the broadcast from the other connectors at the time of the initial connection from one host, the pool is unaware of the others. The "repair" operation causes this to be conducted from each host, and as I guessed, then fixes the SR connectivity for the entire pool. I am speculating that the request for a new connection of this type via XenCenter could perhaps be solved by propagating that initial connection process to the rest of the hosts in the pool and then rescan the SR.
-=Tobias
Hi Tobias,
As for I know, the Dell MD36xx is a dual controller based storage. So it's basically "two computers". Considering this with the added knowledge that I've about TCP networking there's a golden rule that says: only one IP address on a given subnet on a single computer. So I can't use the same network on my case. That's why I use point-to-point links too.
Since I've only one machine acting as a storage server (It's a Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do need four separate networks. FreeNAS is FreeBSD based, and BSD enforces this rule by default. I can't have two IP addresses of the same subnet on same machine, even if I manually configure it.
If this wasn't bad TCP network practice, I would be able to configure the SR via XenCenter, since it will "find" the same network on the others hosts without problem.
About the XenCenter, it really appears to be only a missing function on the software. If I was a good programmer I would try to implement this. But this is beyond my knowledge... :(
Thanks once again,
Vinícius.
No reason to necessarily use four separate subnets. You could have 192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and 2.2 on the other controller (on, for example, an MD36xx that is standard). If however it is something like an MD32xx which recommends four separate subets, then of course, use four. One should follow the vendor's specifications.
I will respond in more detail later when I get a chance, Vinícius, but after a quick read, I agree with your findings from your other email. It would appear that the initial connection is indeed evidently a missing function in XenCenter and perhaps all that is needed to make this work "out of the box". As to the wildcard discovery, it's generally preferred as it allows the host to figure out the path on its own. The only reason I could see not using a wildcard discovery would be if there were any chance for overlaps and ambiguous connections.
Regards,
--Tobias
Hello Dave,
+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+
192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30
192.168.10.2/30
192.168.11.2/30
192.168.20.2/30
192.168.21.2/30
That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0
Cheers,
Vinícius.
Hi,
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked… Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0
I’m really excited because it worked. So let’s me explain what I’ve did.
First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I’d like to understand your configuration better— have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?
Thanks,
Dave
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0
I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/
It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Vinícius Ferrão
2015-02-20 22:53:31 UTC
Permalink
Guys, another important thing that I've observed.

All the iscsiadm and multipath commands are useless. I've attached the same storage just issuing a specific PBD command from the Pool Master. So this simplifies the problem a lot.

I just used this command:

xe pbd-create \
sr-uuid=f8d481b8-be10-922e-b875-91c4d73d341d \
host-uuid=44955863-f4a3-4770-bd7a-f4342dd7925a \
device-config:multiSession="192.168.30.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.31.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" \
device-config:target=192.168.30.1,192.168.31.1 \
device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 \
device-config:targetIQN=* \
device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 \
device-config:port=3260

The device-config attributes are sufficient to create all the connection.

Thanks,
Vinicius.
It may be possible to at least temporarily get this work by forcing the SrRepairAction procedure to be run.
I didn't have time yet, but wanted to compare what it does in sequence vs. what is done by default in the xapi_sr
program to see what is different. As far as XenCenter is concerned, perhaps if multiple IP target addresses are entered in the dialog box (as opposed to a single one or a wildcard entry), then a different means could be triggered to ensure that each host does a proper PBD creation and plug-in for that new SR.
-=Tobias
Post by Vinícius Ferrão
Dave,
I've added the "future" server to our pool. And happened exactly what you said. The configurations were cloned. But obviously it failed to clone the PDB's and the Network configuration for the iSCSI network.
1. The network configuration on added host should be "confirmed" to the user, and not done automatically, so the user can adapt is to specific needs.
2. After this the PDB creation should be done accordingly. If the network topology has changed by the user, so the PDBs must be created with this new topology.
In resume, I've three host at this moment on the pool. So I can do the HA tests once again, and I will report to the list the results. At this moment the only thing that failed is cloning the iSCSI network configurations and plugging the PDBs. Only the bounded NIC0+NIC1 was created on the added host.
Thanks in advance,
Vinícius.
Post by Dave Scott
Regarding Point 3) I believe what is meant is that you should test having more than one SR that you are connecting to using the same IP addresses for the target array. You know it works fine to create one SR, so can you add additional ones the same way and have them all live together in harmony? :-)
As to the sr-create command, the host-uuid is an optional parameter and hence does not need to be named for the pool master or any other. The only required parameters are the name-label and the SR type. Have you tried that to see if the behavior is any different compared to when you do supply a host name?
I believe the underlying Xapi data model can express what you want, but there are 1 or 2 places where Xapi will helpfully create PBDs for you by cloning the master, which you just need to be aware of.
For example you should be able to use something like
$ xe sr-create .. host-uuid=<any host> shared=false
Setting “shared=false” initially should avoid Xapi ‘helpfully’ pre-creating identical PBDs for every host with the same configuration[1]. You should end up with an SR attached to a single host.
$ xe sr-param-set uuid= shared=true
$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...
$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...
[ as an aside, it’s obviously not great that the PBDs you get depend on exactly when you declare the SR is shared. This has happened because the original design had all PBDs being distinct, and the notion of ‘copying the master’ was added relatively late ]
Whenever a new host is added to the pool, Xapi will clone the master’s PBD for you— you would have to destroy it and re-create it. I suppose cloning the master’s configuration is as good a guess as any in this situation :-)
Dave
[1] https://github.com/xapi-project/xen-api/blob/b6f28c52fd9726553968b410e6b8cced5401f385/ocaml/xapi/xapi_sr.ml#L209
-=Tobias
Post by Vinícius Ferrão
Hello guys,
Answering about all the questions and issues. For those who aren't understanding the network topology, I've made a drawing of the solution. The image is here: https://www.dropbox.com/s/6zwerk6igcyqou3/UFRJ-IQ-VirtualPool.jpg?dl=0
192.168.10.1
192.168.11.1
192.168.20.1
192.168.21.1
The convention to this numbering scheme was: 192.168.{XenServerNumber}{InterfaceNumber}.{Storage(1)/Host(2)}.
192.168.10.2 (XenServer #1 - Interface #0)
192.168.11.2 (XenServer #1 - Interface #1)
192.168.20.2 (XenServer #2 - Interface #0)
192.168.21.2 (XenServer #2 - Interface #1)
Hope this completely clarifies the architecture/topology of the iSCSI network.
Restarting both hosts on the same time. [OK]
Restarting the second host. [OK]
Restarting the pool master. [OK]
2. HA is working too. It fences only the damaged host. I even tried a VM Failover. HA happened without any problem and the lost VM has been restarted on other server. To do the HA test, I've power cycled the host, simulated a hardware crash scenario. A cool picture: https://www.dropbox.com/s/g7h7rljk8vh97rg/Screenshot%202015-02-18%2015.45.41.png?dl=0
3. I haven't understood this test. You are asking to create a new LUN and add a new Shared Repository? It's not clear what you're saying with "independent SR".
4. Storage XenMotion works too. I've moved a running VM from local storage (on the Host #2) to the iSCSI pool without any problem. The reverse process works too.
5. I will test this tomorrow, when I will got physical access to the servers (carnival holiday here in BR). But about this specific test, I can't see any problem with plain Xen Motion, since the data is copied over the management network and not the iSCSI network. But I can see a good point when dealing with Storage XenMotion.
Finally about the changes on XenCenter or XenServer/XAPI itself. One thing to prove this point is to create the SR via the CLI. If I would be able to do this, the problem should definitely be on XenCenter. As I said before, I've created a broken Software iSCSI SR via XenCenter and them mapped the iSCSI interfaces, enabled multipath and created the PBD over the CLI.
The iSCSI and Multipath should be done in the physical host. But the PBD creation can be done without any problem on the Pool Master.
device-config: name-label= shared= type=
In this case sm-config and device-config are the problematic ones. Assuming that the host-uuid is always the pool master.
Thanks in advance,
Vinícius.
I’ll add in a few more tests….
1. Host restart. Does the normal PBD plug mechanism work with such a configuration
2. HA. While HA with two hosts isn’t a supported configuration, in this model it might work properly. If you remove the network cables for a single host, does it fence, or do both hosts fence?
3. Multiple SRs. If you have multiple LUNs on the FreeNAS exported and then used as independent SRs, does everything work as it should?
4. Storage XenMotion. With a vdi on one SR presented with the FreeNAS, copy it to any another storage option
5. Motion during path failure. With one path down on a given host, does XenMotion and Storage XenMotion still function correctly?
For those who are questioning the value of such a configuration, this is in essence a “cluster in a box” solution. These configurations were very popular 10-15 years ago as highly available compute infrastructure was quite expensive back then. Today we can accomplish the same things we did back then using virtualization, but “cluster in a box” can always be valuable for those with limited IT skills or for smaller organizations wanting least cost with fewest points of failure.
-tim
Sent: Monday, February 16, 2015 7:24 PM
To: Tobias Kreidl
Subject: Re: [xs-devel] Switchless shared iSCSI Network with /30 links
Tobias,
I've done more tests about the SR creation over XenCenter.
All the process definitely occurs only on the pool master. Even if the specified networks are not part of the networks of the host. The connection is tried on the default route.
192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1
The target IQN is discovered without any problem, since the iSCSI Storage reports the LUNs on any interface.
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 192.168.10.2:55870 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52654 192.168.11.1:3260 TIME_WAIT
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.11.2:52656 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
******************
tcp 0 1 10.7.0.101:56481 192.168.21.1:3260 SYN_SENT
******************
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
******************
tcp 0 1 10.7.0.101:52787 192.168.30.1:3260 SYN_SENT
******************
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
If you're wondering the 192.168.30.1 is reported because in the future we will add another XS host to the pool. The point here is that XenCenter should not lookup for this network since it isn't specified during the Software iSCSI creation. On my understanding it should only lookup for the specified addresses.
tcp 0 48 10.7.0.102:22 192.168.172.1:58046 ESTABLISHED
udp 0 0 192.168.21.2:123 0.0.0.0:*
udp 0 0 192.168.20.2:123 0.0.0.0:*
As you said, perhaps propagating the initial connection through the host will solve the problem without modifying the XenCenter interface. A more sophisticated approach would be query the network-list over XAPI, retrieve the PIFs associated to those networks and them just check the IP addresses on the SR creation request and match them with the equivalent PIFs.
uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa
name-label ( RW): iSCSI #0
PIF-uuids (SRO): 3baa8436-feca-cd04-3173-5002d9ffc66b; 362d63cb-647a-465f-6b81-1b59cd3b1f5d
MTU ( RW): 9000
bridge ( RO): xenbr2
other-config (MRW): automatic: true
default-locking-mode ( RW): unlocked
uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5c:64
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
host-name-label ( RO): xenserver2.local.iq.ufrj.br
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.20.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5a:1c
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c
host-name-label ( RO): xenserver1.local.iq.ufrj.br
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.10.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
So as you can see, we have the IP addresses and the network mask. Which is sufficient to an efficient and precise algorithm of SR creation.
I thinks this clarifies everything we discussed until now. A patch on XenCenter appears to be really viable.
Many thanks,
Vinicius.
Yes, correct. The MD36xx looks like two separate controller units, each acting as a separate device. For a standalone server or some storage devices, that's right that you need a separate subnet for each connection.
Because a single connection isn't seeing the broadcast from the other connectors at the time of the initial connection from one host, the pool is unaware of the others. The "repair" operation causes this to be conducted from each host, and as I guessed, then fixes the SR connectivity for the entire pool. I am speculating that the request for a new connection of this type via XenCenter could perhaps be solved by propagating that initial connection process to the rest of the hosts in the pool and then rescan the SR.
-=Tobias
Hi Tobias,
As for I know, the Dell MD36xx is a dual controller based storage. So it's basically "two computers". Considering this with the added knowledge that I've about TCP networking there's a golden rule that says: only one IP address on a given subnet on a single computer. So I can't use the same network on my case. That's why I use point-to-point links too.
Since I've only one machine acting as a storage server (It's a Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do need four separate networks. FreeNAS is FreeBSD based, and BSD enforces this rule by default. I can't have two IP addresses of the same subnet on same machine, even if I manually configure it.
If this wasn't bad TCP network practice, I would be able to configure the SR via XenCenter, since it will "find" the same network on the others hosts without problem.
About the XenCenter, it really appears to be only a missing function on the software. If I was a good programmer I would try to implement this. But this is beyond my knowledge... :(
Thanks once again,
Vinícius.
No reason to necessarily use four separate subnets. You could have 192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and 2.2 on the other controller (on, for example, an MD36xx that is standard). If however it is something like an MD32xx which recommends four separate subets, then of course, use four. One should follow the vendor's specifications.
I will respond in more detail later when I get a chance, Vinícius, but after a quick read, I agree with your findings from your other email. It would appear that the initial connection is indeed evidently a missing function in XenCenter and perhaps all that is needed to make this work "out of the box". As to the wildcard discovery, it's generally preferred as it allows the host to figure out the path on its own. The only reason I could see not using a wildcard discovery would be if there were any chance for overlaps and ambiguous connections.
Regards,
--Tobias
Hello Dave,
+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+
192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30
192.168.10.2/30
192.168.11.2/30
192.168.20.2/30
192.168.21.2/30
That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0
Cheers,
Vinícius.
Hi,
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked… Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0
I’m really excited because it worked. So let’s me explain what I’ve did.
First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I’d like to understand your configuration better— have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?
Thanks,
Dave
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0
I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/
It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Tobias Kreidl
2015-02-20 23:14:47 UTC
Permalink
When I mentioned running the SrRepairAction procedure, I meant in
conjunction with what is /not/ happening in _XenCenter_ as a follow-up
to fix things.

So then, it would seem the piece remaining to resolve is what is missing
in XenCenter that is not triggered. My guess is that it can perhaps only
perform a connection using one IP address (or wildcarded address) and
hence apparently doesn't understand what to do with anything involving a
multiSession/multihome list?

It would be a useful diagnostic to see what is parsed out and processed
if you pass in the whole multihome list of

192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260

to XenCenter and see what it ends up trying to do with it.

-=Tobias
Post by Vinícius Ferrão
Guys, another important thing that I've observed.
All the iscsiadm and multipath commands are useless. I've attached the same storage just issuing a specific PBD command from the Pool Master. So this simplifies the problem a lot.
xe pbd-create \
sr-uuid=f8d481b8-be10-922e-b875-91c4d73d341d \
host-uuid=44955863-f4a3-4770-bd7a-f4342dd7925a \
device-config:multiSession="192.168.30.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.31.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" \
device-config:target=192.168.30.1,192.168.31.1 \
device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 \
device-config:targetIQN=* \
device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 \
device-config:port=3260
The device-config attributes are sufficient to create all the connection.
Thanks,
Vinicius.
It may be possible to at least temporarily get this work by forcing the SrRepairAction procedure to be run.
I didn't have time yet, but wanted to compare what it does in sequence vs. what is done by default in the xapi_sr
program to see what is different. As far as XenCenter is concerned, perhaps if multiple IP target addresses are entered in the dialog box (as opposed to a single one or a wildcard entry), then a different means could be triggered to ensure that each host does a proper PBD creation and plug-in for that new SR.
-=Tobias
Post by Vinícius Ferrão
Dave,
I've added the "future" server to our pool. And happened exactly what you said. The configurations were cloned. But obviously it failed to clone the PDB's and the Network configuration for the iSCSI network.
1. The network configuration on added host should be "confirmed" to the user, and not done automatically, so the user can adapt is to specific needs.
2. After this the PDB creation should be done accordingly. If the network topology has changed by the user, so the PDBs must be created with this new topology.
In resume, I've three host at this moment on the pool. So I can do the HA tests once again, and I will report to the list the results. At this moment the only thing that failed is cloning the iSCSI network configurations and plugging the PDBs. Only the bounded NIC0+NIC1 was created on the added host.
Thanks in advance,
Vinícius.
Post by Dave Scott
Regarding Point 3) I believe what is meant is that you should test having more than one SR that you are connecting to using the same IP addresses for the target array. You know it works fine to create one SR, so can you add additional ones the same way and have them all live together in harmony? :-)
As to the sr-create command, the host-uuid is an optional parameter and hence does not need to be named for the pool master or any other. The only required parameters are the name-label and the SR type. Have you tried that to see if the behavior is any different compared to when you do supply a host name?
I believe the underlying Xapi data model can express what you want, but there are 1 or 2 places where Xapi will helpfully create PBDs for you by cloning the master, which you just need to be aware of.
For example you should be able to use something like
$ xe sr-create .. host-uuid=<any host> shared=false
Setting “shared=false” initially should avoid Xapi ‘helpfully’ pre-creating identical PBDs for every host with the same configuration[1]. You should end up with an SR attached to a single host.
$ xe sr-param-set uuid= shared=true
$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...
$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...
[ as an aside, it’s obviously not great that the PBDs you get depend on exactly when you declare the SR is shared. This has happened because the original design had all PBDs being distinct, and the notion of ‘copying the master’ was added relatively late ]
Whenever a new host is added to the pool, Xapi will clone the master’s PBD for you— you would have to destroy it and re-create it. I suppose cloning the master’s configuration is as good a guess as any in this situation :-)
Dave
[1] https://github.com/xapi-project/xen-api/blob/b6f28c52fd9726553968b410e6b8cced5401f385/ocaml/xapi/xapi_sr.ml#L209
-=Tobias
Post by Vinícius Ferrão
Hello guys,
Answering about all the questions and issues. For those who aren't understanding the network topology, I've made a drawing of the solution. The image is here: https://www.dropbox.com/s/6zwerk6igcyqou3/UFRJ-IQ-VirtualPool.jpg?dl=0
192.168.10.1
192.168.11.1
192.168.20.1
192.168.21.1
The convention to this numbering scheme was: 192.168.{XenServerNumber}{InterfaceNumber}.{Storage(1)/Host(2)}.
192.168.10.2 (XenServer #1 - Interface #0)
192.168.11.2 (XenServer #1 - Interface #1)
192.168.20.2 (XenServer #2 - Interface #0)
192.168.21.2 (XenServer #2 - Interface #1)
Hope this completely clarifies the architecture/topology of the iSCSI network.
Restarting both hosts on the same time. [OK]
Restarting the second host. [OK]
Restarting the pool master. [OK]
2. HA is working too. It fences only the damaged host. I even tried a VM Failover. HA happened without any problem and the lost VM has been restarted on other server. To do the HA test, I've power cycled the host, simulated a hardware crash scenario. A cool picture: https://www.dropbox.com/s/g7h7rljk8vh97rg/Screenshot%202015-02-18%2015.45.41.png?dl=0
3. I haven't understood this test. You are asking to create a new LUN and add a new Shared Repository? It's not clear what you're saying with "independent SR".
4. Storage XenMotion works too. I've moved a running VM from local storage (on the Host #2) to the iSCSI pool without any problem. The reverse process works too.
5. I will test this tomorrow, when I will got physical access to the servers (carnival holiday here in BR). But about this specific test, I can't see any problem with plain Xen Motion, since the data is copied over the management network and not the iSCSI network. But I can see a good point when dealing with Storage XenMotion.
Finally about the changes on XenCenter or XenServer/XAPI itself. One thing to prove this point is to create the SR via the CLI. If I would be able to do this, the problem should definitely be on XenCenter. As I said before, I've created a broken Software iSCSI SR via XenCenter and them mapped the iSCSI interfaces, enabled multipath and created the PBD over the CLI.
The iSCSI and Multipath should be done in the physical host. But the PBD creation can be done without any problem on the Pool Master.
device-config: name-label= shared= type=
In this case sm-config and device-config are the problematic ones. Assuming that the host-uuid is always the pool master.
Thanks in advance,
Vinícius.
Post by Tim Mackey
I’ll add in a few more tests
.
1. Host restart. Does the normal PBD plug mechanism work with such a configuration
2. HA. While HA with two hosts isn’t a supported configuration, in this model it might work properly. If you remove the network cables for a single host, does it fence, or do both hosts fence?
3. Multiple SRs. If you have multiple LUNs on the FreeNAS exported and then used as independent SRs, does everything work as it should?
4. Storage XenMotion. With a vdi on one SR presented with the FreeNAS, copy it to any another storage option
5. Motion during path failure. With one path down on a given host, does XenMotion and Storage XenMotion still function correctly?
For those who are questioning the value of such a configuration, this is in essence a “cluster in a box” solution. These configurations were very popular 10-15 years ago as highly available compute infrastructure was quite expensive back then. Today we can accomplish the same things we did back then using virtualization, but “cluster in a box” can always be valuable for those with limited IT skills or for smaller organizations wanting least cost with fewest points of failure.
-tim
Sent: Monday, February 16, 2015 7:24 PM
To: Tobias Kreidl
Subject: Re: [xs-devel] Switchless shared iSCSI Network with /30 links
Tobias,
I've done more tests about the SR creation over XenCenter.
All the process definitely occurs only on the pool master. Even if the specified networks are not part of the networks of the host. The connection is tried on the default route.
192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1
The target IQN is discovered without any problem, since the iSCSI Storage reports the LUNs on any interface.
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 192.168.10.2:55870 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52654 192.168.11.1:3260 TIME_WAIT
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.11.2:52656 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
******************
tcp 0 1 10.7.0.101:56481 192.168.21.1:3260 SYN_SENT
******************
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
******************
tcp 0 1 10.7.0.101:52787 192.168.30.1:3260 SYN_SENT
******************
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
If you're wondering the 192.168.30.1 is reported because in the future we will add another XS host to the pool. The point here is that XenCenter should not lookup for this network since it isn't specified during the Software iSCSI creation. On my understanding it should only lookup for the specified addresses.
tcp 0 48 10.7.0.102:22 192.168.172.1:58046 ESTABLISHED
udp 0 0 192.168.21.2:123 0.0.0.0:*
udp 0 0 192.168.20.2:123 0.0.0.0:*
As you said, perhaps propagating the initial connection through the host will solve the problem without modifying the XenCenter interface. A more sophisticated approach would be query the network-list over XAPI, retrieve the PIFs associated to those networks and them just check the IP addresses on the SR creation request and match them with the equivalent PIFs.
uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa
name-label ( RW): iSCSI #0
PIF-uuids (SRO): 3baa8436-feca-cd04-3173-5002d9ffc66b; 362d63cb-647a-465f-6b81-1b59cd3b1f5d
MTU ( RW): 9000
bridge ( RO): xenbr2
other-config (MRW): automatic: true
default-locking-mode ( RW): unlocked
uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5c:64
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
host-name-label ( RO): xenserver2.local.iq.ufrj.br
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.20.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5a:1c
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c
host-name-label ( RO): xenserver1.local.iq.ufrj.br
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.10.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
So as you can see, we have the IP addresses and the network mask. Which is sufficient to an efficient and precise algorithm of SR creation.
I thinks this clarifies everything we discussed until now. A patch on XenCenter appears to be really viable.
Many thanks,
Vinicius.
Yes, correct. The MD36xx looks like two separate controller units, each acting as a separate device. For a standalone server or some storage devices, that's right that you need a separate subnet for each connection.
Because a single connection isn't seeing the broadcast from the other connectors at the time of the initial connection from one host, the pool is unaware of the others. The "repair" operation causes this to be conducted from each host, and as I guessed, then fixes the SR connectivity for the entire pool. I am speculating that the request for a new connection of this type via XenCenter could perhaps be solved by propagating that initial connection process to the rest of the hosts in the pool and then rescan the SR.
-=Tobias
Hi Tobias,
As for I know, the Dell MD36xx is a dual controller based storage. So it's basically "two computers". Considering this with the added knowledge that I've about TCP networking there's a golden rule that says: only one IP address on a given subnet on a single computer. So I can't use the same network on my case. That's why I use point-to-point links too.
Since I've only one machine acting as a storage server (It's a Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do need four separate networks. FreeNAS is FreeBSD based, and BSD enforces this rule by default. I can't have two IP addresses of the same subnet on same machine, even if I manually configure it.
If this wasn't bad TCP network practice, I would be able to configure the SR via XenCenter, since it will "find" the same network on the others hosts without problem.
About the XenCenter, it really appears to be only a missing function on the software. If I was a good programmer I would try to implement this. But this is beyond my knowledge... :(
Thanks once again,
Vinícius.
No reason to necessarily use four separate subnets. You could have 192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and 2.2 on the other controller (on, for example, an MD36xx that is standard). If however it is something like an MD32xx which recommends four separate subets, then of course, use four. One should follow the vendor's specifications.
I will respond in more detail later when I get a chance, Vinícius, but after a quick read, I agree with your findings from your other email. It would appear that the initial connection is indeed evidently a missing function in XenCenter and perhaps all that is needed to make this work "out of the box". As to the wildcard discovery, it's generally preferred as it allows the host to figure out the path on its own. The only reason I could see not using a wildcard discovery would be if there were any chance for overlaps and ambiguous connections.
Regards,
--Tobias
Hello Dave,
+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+
192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30
192.168.10.2/30
192.168.11.2/30
192.168.20.2/30
192.168.21.2/30
That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0
Cheers,
Vinícius.
Hi,
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked
 Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0
I’m really excited because it worked. So let’s me explain what I’ve did.
First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I’d like to understand your configuration better— have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?
Thanks,
Dave
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0
I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/
It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Vinícius Ferrão
2015-02-21 03:49:40 UTC
Permalink
Any ideia on how to test this? I tried to create the pool for the first time using the string with all the six IP addresses, but XenCenter just fails without and error message, just an single X appears. The only thing I can provide at this moment is the /var/log/xensource.log; /var/log/audit.log; /var/log/messages and /var/log/SMlog file during the IQN/LUN phase, it's on the end of this message. The /var/log/SMlog appears to be the most valuable one.

I'm really dedicated on helping solving this issue, since I'm delaying the implementation of this farm just to keep the test scenario live!

Thanks,

PS: The logs...

*****/var/log/xensource.log*****
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|Async.SR.create R:b933cd75031b|dispatcher] spawning a new thread to handle the current task (trackid=8c8347a3f363fd40a9bd41eb1705050f)
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|audit] SR.create: name label = '__gui__'
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] SR.create name_label=__gui__ sm_config=[ ]
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|Async.SR.create R:b933cd75031b|storage_access] SR d403d0fc-4a14-7f99-fe92-dfc6301ed764 will be implemented by /services/SM/lvmoiscsi in VM OpaqueRef:47f2c383-de3e-730e-9836-d652d5eae646
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|mux] register SR d403d0fc-4a14-7f99-fe92-dfc6301ed764 (currently-registered = [ f31651b7-4219-dfe5-d871-4748c05e900c, ee47042b-76bf-5f62-2879-0457db8e892d, 12945887-3593-6c24-0b77-2efd62949415, d403d0fc-4a14-7f99-fe92-dfc6301ed764, 551b026c-6b94-5e08-ae67-3c76c15b8b44 ])
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|dummytaskhelper] task SR.create D:ebc6f0ef8713 created by task R:b933cd75031b
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|sm] SM lvmoiscsi sr_create sr=OpaqueRef:383eb943-faa8-1946-f169-6626b1996fe7 size=0
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Session.create trackid=70acc8c1d9618c45c2f07860af213730 pool=false uname= originator= is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|mscgen] xapi=>xapi [label="(XML)"];
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14339 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:7e7d99be1ccd created by task D:208b3f98f095
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|mscgen] smapiv2=>smapiv1 [label="sr_create"];
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:e234520429b7|api_effect] task.add_to_other_config
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|xenops] Event on VM 6dd2a08f-43a3-4f0c-b6c4-7e040829dc6a; resident_here = true
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|mscgen] xapi=>xenops [label="VM.exists"];
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|xenops] kernel is not in the whitelist: ignoring
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|mscgen] xapi=>xapi [label="(XML)"];
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14343 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:event.from D:ff5a707a4074 created by task D:4b65ef16de2c
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:7d79581bbf17|api_effect] task.add_to_other_config
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14344 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:27e54f4f8e11 created by task D:ebc6f0ef8713
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14345 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:fac3f1373b6e created by task D:ebc6f0ef8713
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14346 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:1713380916eb created by task D:ebc6f0ef8713
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:538ce8692691|api_effect] task.add_to_other_config
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Raised at sm_exec.ml:212.7-69 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Session.destroy trackid=70acc8c1d9618c45c2f07860af213730
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|backtrace] Raised at pervasiveext.ml:26.22-25 -> server_helpers.ml:76.11-23
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|dispatcher] Server_helpers.exec exception_handler: Got exception SR_BACKEND_FAILURE_96: [ ; The request is missing or has an incorrect target IQN parameter; <?xml version="1.0" ?> <iscsi-target-iqns> <TGT> <Index> 0 </Index> <IPAddress> 192.168.10.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 1 </Index> <IPAddress> 192.168.11.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 2 </Index> <IPAddress> 192.168.20.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 3 </Index> <IPAddress> 192.168.21.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 4 </Index> <IPAddress> 192.168.30.1 </IPAddress>
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|dispatcher] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|backtrace] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|backtrace] Raised at storage_access.ml:161.15-44 -> server_helpers.ml:76.11-23
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|dispatcher] Server_helpers.exec exception_handler: Got exception INTERNAL_ERROR: [ Storage_interface.Backend_error(_) ]
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|dispatcher] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|backtrace] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] Raised at pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [error|xenserver1|14336|Async.SR.create R:b933cd75031b|storage_access] Re-raising as SR_BACKEND_FAILURE_96 [ ; The request is missing or has an incorrect target IQN parameter; <?xml version="1.0" ?> <iscsi-target-iqns> <TGT> <Index> 0 </Index> <IPAddress> 192.168.10.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 1 </Index> <IPAddress> 192.168.11.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 2 </Index> <IPAddress> 192.168.20.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 3 </Index> <IPAddress> 192.168.21.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 4 </Index> <IPAddress> 192.168.30.1 </IPAddress> <TargetIQN>iqn.2015-01.b
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|backtrace] Raised at xapi_sr.ml:225.9-10 -> server.ml:21898.82-267 -> rbac.ml:229.16-23
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|backtrace] Raised at rbac.ml:238.10-15 -> server_helpers.ml:79.11-41
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|dispatcher] Server_helpers.exec exception_handler: Got exception SR_BACKEND_FAILURE_96: [ ; The request is missing or has an incorrect target IQN parameter; <?xml version="1.0" ?> <iscsi-target-iqns> <TGT> <Index> 0 </Index> <IPAddress> 192.168.10.1 </IPAddress> <TargetIQN>iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 1 </Index> <IPAddress> 192.168.11.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 2 </Index> <IPAddress>192.168.20.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 3 </Index> <IPAddress> 192.168.21.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 4 </Index> <IPAddress> 192.168.30.1 </IPAdd
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|dispatcher] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|backtrace] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336||xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336||xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|Async.SR.create R:202cf9100203|dispatcher] spawning a new thread to handle the current task (trackid=8c8347a3f363fd40a9bd41eb1705050f)
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|audit] SR.create: name label = '__gui__'
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] SR.create name_label=__gui__ sm_config=[ ]
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|Async.SR.create R:202cf9100203|storage_access] SR 207535ee-c14c-7b48-3869-96318c1c7877 will be implemented by /services/SM/lvmoiscsi in VM OpaqueRef:47f2c383-de3e-730e-9836-d652d5eae646
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|mux] register SR 207535ee-c14c-7b48-3869-96318c1c7877 (currently-registered = [ f31651b7-4219-dfe5-d871-4748c05e900c, ee47042b-76bf-5f62-2879-0457db8e892d, 207535ee-c14c-7b48-3869-96318c1c7877, 12945887-3593-6c24-0b77-2efd62949415, d403d0fc-4a14-7f99-fe92-dfc6301ed764, 551b026c-6b94-5e08-ae67-3c76c15b8b44 ])
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|dummytaskhelper] task SR.create D:7e6802e5fed5 created by task R:202cf9100203
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|sm] SM lvmoiscsi sr_create sr=OpaqueRef:5591eb47-1a35-9321-6aa4-bd8beb748cbf size=0
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Session.create trackid=836c7784c6fd5a42124fb4bd64aa2cd5 pool=false uname= originator= is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|mscgen] xapi=>xapi [label="(XML)"];
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14357 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:c14a1f2ccf30 created by task D:4a4af7d945d9
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|mscgen] smapiv2=>smapiv1 [label="sr_create"];
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:6f63db465b85|api_effect] task.add_to_other_config
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|xenops] Event on VM 6dd2a08f-43a3-4f0c-b6c4-7e040829dc6a; resident_here = true
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|mscgen] xapi=>xenops [label="VM.exists"];
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|xenops] kernel is not in the whitelist: ignoring
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|mscgen] xapi=>xapi [label="(XML)"];
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14359 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:event.from D:dd4a82303df3 created by task D:4b65ef16de2c
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:454990b8d01f|api_effect] task.add_to_other_config
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14360 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:513472992344 created by task D:7e6802e5fed5
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14361 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:4cb6c60153d2 created by task D:7e6802e5fed5
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14362 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:901d21392528 created by task D:7e6802e5fed5
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:f7f97c5c1dec|api_effect] task.add_to_other_config
Feb 21 01:41:17 xenserver1 xapi: [debug|xenserver1|14365 INET 0.0.0.0:80|Get RRD updates. D:8f822f0bc4c4|xapi] hand_over_connection GET /rrd_updates to /var/xapi/xcp-rrdd.forwarded
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Raised at forkhelpers.ml:187.30-76 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Raised at sm_exec.ml:190.10-100 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Raised at pervasiveext.ml:26.22-25 -> sm_exec.ml:173.23-1023 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [ info|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Session.destroy trackid=836c7784c6fd5a42124fb4bd64aa2cd5
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|backtrace] Raised at pervasiveext.ml:26.22-25 -> server_helpers.ml:76.11-23
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|dispatcher] Server_helpers.exec exception_handler: Got exception SR_BACKEND_FAILURE: [ non-zero exit; ; Traceback (most recent call last): File "/opt/xensource/sm/LVMoISCSISR", line 582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File "/opt/xensource/sm/SRCommand.py", line 343, in run sr = driver(cmd, cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line 142, in __init__ self.load(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line 156, in load self.iscsi = self.iscsiSRs[0] IndexError: list index out of range ]
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|dispatcher] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|backtrace] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|backtrace] Raised at storage_access.ml:161.15-44 -> server_helpers.ml:76.11-23
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|dispatcher] Server_helpers.exec exception_handler: Got exception INTERNAL_ERROR: [ Storage_interface.Backend_error(_) ]
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|dispatcher] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|backtrace] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] Raised at pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [error|xenserver1|14352|Async.SR.create R:202cf9100203|storage_access] Re-raising as SR_BACKEND_FAILURE [ non-zero exit; ; Traceback (most recent call last): File "/opt/xensource/sm/LVMoISCSISR", line 582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File "/opt/xensource/sm/SRCommand.py", line 343, in run sr = driver(cmd, cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line 142, in __init__ self.load(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line 156, in load self.iscsi = self.iscsiSRs[0] IndexError: list index out of range ]
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|backtrace] Raised at xapi_sr.ml:225.9-10 -> server.ml:21898.82-267 -> rbac.ml:229.16-23
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|backtrace] Raised at rbac.ml:238.10-15 -> server_helpers.ml:79.11-41
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|dispatcher] Server_helpers.exec exception_handler: Got exception SR_BACKEND_FAILURE: [ non-zero exit; ; Traceback (most recent call last): File "/opt/xensource/sm/LVMoISCSISR", line 582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File "/opt/xensource/sm/SRCommand.py", line 343, in run sr = driver(cmd, cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line 142, in __init__ self.load(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line 156, in load self.iscsi = self.iscsiSRs[0] IndexError: list index out of range ]
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|dispatcher] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|backtrace] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352||xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352||xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0 monitor|main|rrdd_stats] system stats: MemTotal: 3959600 KiB; MemFree: 3262140 KiB; Buffered: 33832 KiB; Cached: 291048 KiB; SwapTotal: 524284 KiB; SwapFree: 524284 KiB
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0 monitor|main|rrdd_stats] Clock drift: -0
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0 monitor|main|rrdd_stats] xcp-rrdd stats (n = 1): size: 155052 KiB; rss: 8368 KiB; data: 71620 KiB; stack: 136 KiB
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0 monitor|main|rrdd_stats] xapi stats (n = 2): size: 604328 KiB; rss: 101844 KiB; data: 451340 KiB; stack: 272 KiB
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0 monitor|main|rrdd_stats] xenopsd stats (n = 1): size: 278248 KiB; rss: 8308 KiB; data: 193692 KiB; stack: 136 KiB




***** /var/log/audit.log *****
Feb 21 01:41:01 xenserver1 xapi: [20150221T03:41:01.707Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:ee0292d7107f|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' '577acdd7-2b8c-b177-bf45-f8422c4ed7ca' 'OpaqueRef:b933cd75-031b-6ff0-5a0a-2341507af648')))
Feb 21 01:41:01 xenserver1 xapi: [20150221T03:41:01.747Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:ef6d521270fb|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' '577acdd7-2b8c-b177-bf45-f8422c4ed7ca' 'OpaqueRef:b933cd75-031b-6ff0-5a0a-2341507af648') ('value' '' 'e1aaff51-0da2-2745-cf25-ab034aea8a25' 'OpaqueRef:85f25443-719e-105d-35aa-a1c1043fdd8e')))
Feb 21 01:41:01 xenserver1 xapi: [20150221T03:41:01.787Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:43071305947a|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' '577acdd7-2b8c-b177-bf45-f8422c4ed7ca' 'OpaqueRef:b933cd75-031b-6ff0-5a0a-2341507af648')))
Feb 21 01:41:01 xenserver1 xapi: [20150221T03:41:01.885Z|audit|xenserver1|14336|Async.SR.create R:b933cd75031b|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:SR_BACKEND_FAILURE_96: [ ; The request is missing or has an incorrect target IQN parameter; <?xml version=\"1.0\" ?> <iscsi-target-iqns> <TGT> <Index> 0 </Index> <IPAddress> 192.168.10.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 1 </Index> <IPAddress> 192.168.11.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 2 </Index> <IPAddress> 192.168.20.1 </IPAddress> <TargetIQN>iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 3 </Index> <IPAddress> 192.168.21.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index>
Feb 21 01:41:02 xenserver1 xapi: [20150221T03:41:02.883Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.destroy D:5804968b2ed6|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.destroy' (('self' 'Async.SR.create' '577acdd7-2b8c-b177-bf45-f8422c4ed7ca' 'OpaqueRef:b933cd75-031b-6ff0-5a0a-2341507af648')))
Feb 21 01:41:06 xenserver1 xapi: [20150221T03:41:06.598Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:a09ea039fcd0|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' 'a0d8e973-9995-fb15-532e-f36ac5184c28' 'OpaqueRef:202cf910-0203-a943-9054-8d0e25a2b4a7')))
Feb 21 01:41:06 xenserver1 xapi: [20150221T03:41:06.638Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:b4dd3f0d946a|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' 'a0d8e973-9995-fb15-532e-f36ac5184c28' 'OpaqueRef:202cf910-0203-a943-9054-8d0e25a2b4a7') ('value' '' 'e1aaff51-0da2-2745-cf25-ab034aea8a25' 'OpaqueRef:85f25443-719e-105d-35aa-a1c1043fdd8e')))
Feb 21 01:41:06 xenserver1 xapi: [20150221T03:41:06.679Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:9b0a16a91401|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' 'a0d8e973-9995-fb15-532e-f36ac5184c28' 'OpaqueRef:202cf910-0203-a943-9054-8d0e25a2b4a7')))
Feb 21 01:41:17 xenserver1 xapi: [20150221T03:41:18.001Z|audit|xenserver1|14365 INET 0.0.0.0:80|handler:http/rrd_updates D:3d1bf1a8b00c|audit] ('trackid=bf92de30818eb264b5673ffe734ad369' 'LOCAL_SUPERUSER' '' 'ALLOWED' 'OK' 'HTTP' 'http/rrd_updates' ())
Feb 21 01:41:35 xenserver1 xapi: [20150221T03:41:35.155Z|audit|xenserver1|14352|Async.SR.create R:202cf9100203|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:SR_BACKEND_FAILURE: [ non-zero exit; ; Traceback (most recent call last): File \"/opt/xensource/sm/LVMoISCSISR\", line 582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File \"/opt/xensource/sm/SRCommand.py\", line 343, in run sr = driver(cmd, cmd.sr_uuid) File \"/opt/xensource/sm/SR.py\", line 142, in __init__ self.load(sr_uuid) File \"/opt/xensource/sm/LVMoISCSISR\", line 156, in load self.iscsi = self.iscsiSRs[0] IndexError: list index out of range ]' 'API' 'SR.create' (('host' 'xenserver1.local.iq.ufrj.br' 'c8927969-b7bf-4a0f-9725-035550381d9c' 'OpaqueRef:03239cf1-76cf-9c86-b927-56ea27cb6b94') ('name_label' '__gui__' '' '') ('name_description' 'SHOULD NEVER BE CREATED' '' '')))
Feb 21 01:41:35 xenserver1 xapi: [20150221T03:41:35.707Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.destroy D:7ff3ff7db08a|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.destroy' (('self' 'Async.SR.create' 'a0d8e973-9995-fb15-532e-f36ac5184c28' 'OpaqueRef:202cf910-0203-a943-9054-8d0e25a2b4a7')))





***** /var/log/messages *****
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|Async.SR.create R:b933cd75031b|dispatcher] spawning a new thread to handle the current task (trackid=8c8347a3f363fd40a9bd41eb1705050f)
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|Async.SR.create R:b933cd75031b|storage_access] SR d403d0fc-4a14-7f99-fe92-dfc6301ed764 will be implemented by /services/SM/lvmoiscsi in VM OpaqueRef:47f2c383-de3e-730e-9836-d652d5eae646
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Session.create trackid=70acc8c1d9618c45c2f07860af213730 pool=false uname= originator= is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Session.destroy trackid=70acc8c1d9618c45c2f07860af213730
Feb 21 01:41:01 xenserver1 xapi: [error|xenserver1|14336|Async.SR.create R:b933cd75031b|storage_access] Re-raising as SR_BACKEND_FAILURE_96 [ ; The request is missing or has an incorrect target IQN parameter; <?xml version="1.0" ?> <iscsi-target-iqns> <TGT> <Index> 0 </Index> <IPAddress> 192.168.10.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 1 </Index> <IPAddress> 192.168.11.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 2 </Index> <IPAddress> 192.168.20.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 3 </Index> <IPAddress> 192.168.21.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 4 </Index> <IPAddress> 192.168.30.1 </IPAddress> <TargetIQN>iqn.2015-01.b
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|Async.SR.create R:202cf9100203|dispatcher] spawning a new thread to handle the current task (trackid=8c8347a3f363fd40a9bd41eb1705050f)
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|Async.SR.create R:202cf9100203|storage_access] SR 207535ee-c14c-7b48-3869-96318c1c7877 will be implemented by /services/SM/lvmoiscsi in VM OpaqueRef:47f2c383-de3e-730e-9836-d652d5eae646
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Session.create trackid=836c7784c6fd5a42124fb4bd64aa2cd5 pool=false uname= originator= is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05155|bond|INFO|bond bond0: shift 21kB of load (with hash 35) from eth1 to eth0 (now carrying 233kB and 94kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05156|bond|INFO|bond bond0: shift 224kB of load (with hash 39) from eth1 to eth0 (now carrying 9kB and 318kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05157|bond|INFO|bond bond0: shift 0kB of load (with hash 9) from eth0 to eth1 (now carrying 318kB and 9kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05158|bond|INFO|bond bond0: shift 1kB of load (with hash 12) from eth0 to eth1 (now carrying 317kB and 10kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05159|bond|INFO|bond bond0: shift 1kB of load (with hash 14) from eth0 to eth1 (now carrying 316kB and 11kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05160|bond|INFO|bond bond0: shift 0kB of load (with hash 20) from eth0 to eth1 (now carrying 315kB and 11kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05161|bond|INFO|bond bond0: shift 0kB of load (with hash 21) from eth0 to eth1 (now carrying 315kB and 12kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05162|bond|INFO|bond bond0: shift 0kB of load (with hash 25) from eth0 to eth1 (now carrying 315kB and 12kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05163|bond|INFO|bond bond0: shift 15kB of load (with hash 46) from eth0 to eth1 (now carrying 300kB and 27kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05164|bond|INFO|bond bond0: shift 2kB of load (with hash 70) from eth0 to eth1 (now carrying 297kB and 30kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05165|bond|INFO|bond bond0: shift 1kB of load (with hash 84) from eth0 to eth1 (now carrying 295kB and 32kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05166|bond|INFO|bond bond0: shift 0kB of load (with hash 94) from eth0 to eth1 (now carrying 295kB and 32kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05167|bond|INFO|bond bond0: shift 2kB of load (with hash 112) from eth0 to eth1 (now carrying 293kB and 34kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05168|bond|INFO|bond bond0: shift 28kB of load (with hash 118) from eth0 to eth1 (now carrying 265kB and 62kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05169|bond|INFO|bond bond0: shift 5kB of load (with hash 160) from eth0 to eth1 (now carrying 259kB and 68kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05170|bond|INFO|bond bond0: shift 5kB of load (with hash 171) from eth0 to eth1 (now carrying 254kB and 73kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05171|bond|INFO|bond bond0: shift 3kB of load (with hash 206) from eth0 to eth1 (now carrying 251kB and 76kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05172|bond|INFO|bond bond0: shift 2kB of load (with hash 254) from eth0 to eth1 (now carrying 249kB and 78kB load, respectively)
Feb 21 01:41:28 xenserver1 ovs-vswitchd: ovs|05173|ofproto|INFO|xapi0: 29 flow_mods in the last 59 s (29 adds)
Feb 21 01:41:35 xenserver1 fe: 2498 (/opt/xensource/sm/LVMoISCSISR <methodCall><methodName>sr_create</methodName><...) exited with code 1
Feb 21 01:41:35 xenserver1 xapi: [ info|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Session.destroy trackid=836c7784c6fd5a42124fb4bd64aa2cd5
Feb 21 01:41:35 xenserver1 xapi: [error|xenserver1|14352|Async.SR.create R:202cf9100203|storage_access] Re-raising as SR_BACKEND_FAILURE [ non-zero exit; ; Traceback (most recent call last): File "/opt/xensource/sm/LVMoISCSISR", line 582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File "/opt/xensource/sm/SRCommand.py", line 343, in run sr = driver(cmd, cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line 142, in __init__ self.load(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line 156, in load self.iscsi = self.iscsiSRs[0] IndexError: list index out of range ]
Feb 21 01:41:56 xenserver1 xenstored: D3.14 rm attr/eth0
Feb 21 01:41:56 xenserver1 xenstored: D3 write attr/eth0/ip 10.7.0.243
Feb 21 01:41:56 xenserver1 xenstored: D3 write attr/eth0/ipv6/0/addr fe80::484b:acff:fe91:5b19



***** /var/log/SMlog *****
[***@xenserver1 log]# grep "Feb 21 01:41" /var/log/SMlog
Feb 21 01:41:01 xenserver1 SM: [2444] _testHost: Testing host/port: 192.168.10.1,3260
Feb 21 01:41:01 xenserver1 SM: [2444] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] Raising exception [96, The request is missing or has an incorrect target IQN parameter]
Feb 21 01:41:06 xenserver1 SM: [2498] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] Raising exception [202, General backend error [opterr=rc: 21, stdout: , stderr: iscsiadm: No active sessions.
Feb 21 01:41:06 xenserver1 SM: [2498] ]]
Feb 21 01:41:06 xenserver1 SM: [2498] ['iscsiadm', '-m', 'session'] failed with (u'General backend error [opterr=rc: 21, stdout: , stderr: iscsiadm: No active sessions.\n]',)
Feb 21 01:41:06 xenserver1 SM: [2498] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] ['ls', '/sys/class/scsi_host', '-1', '--color=never']
Feb 21 01:41:06 xenserver1 SM: [2498] pread SUCCESS
Feb 21 01:41:06 xenserver1 SM: [2498] []
Feb 21 01:41:06 xenserver1 SM: [2498] PATHDICT: key 192.168.10.1:3260: {'path': '/dev/iscsi/*/192.168.10.1:3260', 'ipaddr': '192.168.10.1', 'port': 3260L}
Feb 21 01:41:06 xenserver1 SM: [2498] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] Discovery for IP 192.168.10.1 returned [('192.168.10.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.11.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.20.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.21.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.30.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.31.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1')]
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.10.1,3260
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.11.1,3260
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.20.1,3260
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.20.1) - (113, 'No route to host')
Feb 21 01:41:06 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:06 xenserver1 SM: [2498] Target Not reachable: (192.168.20.1:3260)
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.21.1,3260
Feb 21 01:41:07 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.21.1) - (113, 'No route to host')
Feb 21 01:41:07 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:07 xenserver1 SM: [2498] Target Not reachable: (192.168.21.1:3260)
Feb 21 01:41:07 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.30.1,3260
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.30.1) - timed out
Feb 21 01:41:09 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:09 xenserver1 SM: [2498] Target Not reachable: (192.168.30.1:3260)
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.31.1,3260
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.31.1) - (113, 'No route to host')
Feb 21 01:41:09 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:09 xenserver1 SM: [2498] Target Not reachable: (192.168.31.1:3260)
Feb 21 01:41:09 xenserver1 SM: [2498] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:09 xenserver1 SM: [2498] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:09 xenserver1 SM: [2498] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:09 xenserver1 SM: [2498] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:09 xenserver1 SM: [2498] Discovery for IP 192.168.11.1 returned [('192.168.10.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.11.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.20.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.21.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.30.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.31.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1')]
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.10.1,3260
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.11.1,3260
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.20.1,3260
Feb 21 01:41:11 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.20.1) - timed out
Feb 21 01:41:11 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:11 xenserver1 SM: [2498] Target Not reachable: (192.168.20.1:3260)
Feb 21 01:41:11 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.21.1,3260
Feb 21 01:41:11 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.21.1) - (113, 'No route to host')
Feb 21 01:41:11 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:11 xenserver1 SM: [2498] Target Not reachable: (192.168.21.1:3260)
Feb 21 01:41:11 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.30.1,3260
Feb 21 01:41:12 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.30.1) - (113, 'No route to host')
Feb 21 01:41:12 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:12 xenserver1 SM: [2498] Target Not reachable: (192.168.30.1:3260)
Feb 21 01:41:12 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.31.1,3260
Feb 21 01:41:14 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.31.1) - timed out
Feb 21 01:41:14 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:14 xenserver1 SM: [2498] Target Not reachable: (192.168.31.1:3260)
Feb 21 01:41:15 xenserver1 SM: [2498] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:15 xenserver1 SM: [2498] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:35 xenserver1 SM: [2498] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:35 xenserver1 SM: [2498] Raising exception [202, General backend error [opterr=rc: 21, stdout: , stderr: iscsiadm: cannot make connection to 192.168.20.1: No route to host
Feb 21 01:41:35 xenserver1 SM: [2498] iscsiadm: cannot make connection to 192.168.20.1: No route to host
Feb 21 01:41:35 xenserver1 last message repeated 3 times
Feb 21 01:41:35 xenserver1 SM: [2498] iscsiadm: connect to 192.168.20.1 timed out
Feb 21 01:41:35 xenserver1 SM: [2498] iscsiadm: connection login retries (reopen_max) 5 exceeded
Feb 21 01:41:35 xenserver1 SM: [2498] iscsiadm: No portals found
Feb 21 01:41:35 xenserver1 SM: [2498] ]]
Feb 21 01:41:35 xenserver1 SM: [2498] Raising exception [68, ISCSI login failed, verify CHAP credentials]
Feb 21 01:41:35 xenserver1 SM: [2498] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:35 xenserver1 SM: [2498] ***** LVHDoISCSISR.load: EXCEPTION SR.SROSError, ISCSI login failed, verify CHAP credentials
Feb 21 01:41:35 xenserver1 SM: [2498] File "/opt/xensource/sm/LVMoISCSISR", line 119, in load
Feb 21 01:41:35 xenserver1 SM: [2498] map = iscsilib.discovery(tgt_ip,iscsi.port,iscsi.chapuser,iscsi.chappassword,targetIQN=IQN)
Feb 21 01:41:35 xenserver1 SM: [2498] File "/opt/xensource/sm/iscsilib.py", line 156, in discovery
Feb 21 01:41:35 xenserver1 SM: [2498] raise xs_errors.XenError('ISCSILogin')
Feb 21 01:41:35 xenserver1 SM: [2498] File "/opt/xensource/sm/xs_errors.py", line 52, in __init__
Feb 21 01:41:35 xenserver1 SM: [2498] raise SR.SROSError(errorcode, errormessage)
Feb 21 01:41:35 xenserver1 SM: [2498]
Feb 21 01:41:54 xenserver1 updatempppathd: [7617] The garbage collection routine returned: 0
When I mentioned running the SrRepairAction procedure, I meant in conjunction with what is not happening in XenCenter as a follow-up to fix things.
So then, it would seem the piece remaining to resolve is what is missing in XenCenter that is not triggered. My guess is that it can perhaps only perform a connection using one IP address (or wildcarded address) and hence apparently doesn't understand what to do with anything involving a multiSession/multihome list?
It would be a useful diagnostic to see what is parsed out and processed if you pass in the whole multihome list of
192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260
to XenCenter and see what it ends up trying to do with it.
-=Tobias
Post by Vinícius Ferrão
Guys, another important thing that I've observed.
All the iscsiadm and multipath commands are useless. I've attached the same storage just issuing a specific PBD command from the Pool Master. So this simplifies the problem a lot.
xe pbd-create \
sr-uuid=f8d481b8-be10-922e-b875-91c4d73d341d \
host-uuid=44955863-f4a3-4770-bd7a-f4342dd7925a \
device-config:multiSession="192.168.30.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.31.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" \
device-config:target=192.168.30.1,192.168.31.1 \
device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 \
device-config:targetIQN=* \
device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 \
device-config:port=3260
The device-config attributes are sufficient to create all the connection.
Thanks,
Vinicius.
It may be possible to at least temporarily get this work by forcing the SrRepairAction procedure to be run.
I didn't have time yet, but wanted to compare what it does in sequence vs. what is done by default in the xapi_sr
program to see what is different. As far as XenCenter is concerned, perhaps if multiple IP target addresses are entered in the dialog box (as opposed to a single one or a wildcard entry), then a different means could be triggered to ensure that each host does a proper PBD creation and plug-in for that new SR.
-=Tobias
Post by Vinícius Ferrão
Dave,
I've added the "future" server to our pool. And happened exactly what you said. The configurations were cloned. But obviously it failed to clone the PDB's and the Network configuration for the iSCSI network.
1. The network configuration on added host should be "confirmed" to the user, and not done automatically, so the user can adapt is to specific needs.
2. After this the PDB creation should be done accordingly. If the network topology has changed by the user, so the PDBs must be created with this new topology.
In resume, I've three host at this moment on the pool. So I can do the HA tests once again, and I will report to the list the results. At this moment the only thing that failed is cloning the iSCSI network configurations and plugging the PDBs. Only the bounded NIC0+NIC1 was created on the added host.
Thanks in advance,
Vinícius.
Post by Dave Scott
Regarding Point 3) I believe what is meant is that you should test having more than one SR that you are connecting to using the same IP addresses for the target array. You know it works fine to create one SR, so can you add additional ones the same way and have them all live together in harmony? :-)
As to the sr-create command, the host-uuid is an optional parameter and hence does not need to be named for the pool master or any other. The only required parameters are the name-label and the SR type. Have you tried that to see if the behavior is any different compared to when you do supply a host name?
I believe the underlying Xapi data model can express what you want, but there are 1 or 2 places where Xapi will helpfully create PBDs for you by cloning the master, which you just need to be aware of.
For example you should be able to use something like
$ xe sr-create .. host-uuid=<any host> shared=false
Setting “shared=false” initially should avoid Xapi ‘helpfully’ pre-creating identical PBDs for every host with the same configuration[1]. You should end up with an SR attached to a single host.
$ xe sr-param-set uuid= shared=true
$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...
$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...
[ as an aside, it’s obviously not great that the PBDs you get depend on exactly when you declare the SR is shared. This has happened because the original design had all PBDs being distinct, and the notion of ‘copying the master’ was added relatively late ]
Whenever a new host is added to the pool, Xapi will clone the master’s PBD for you— you would have to destroy it and re-create it. I suppose cloning the master’s configuration is as good a guess as any in this situation :-)
Dave
[1] https://github.com/xapi-project/xen-api/blob/b6f28c52fd9726553968b410e6b8cced5401f385/ocaml/xapi/xapi_sr.ml#L209 <https://github.com/xapi-project/xen-api/blob/b6f28c52fd9726553968b410e6b8cced5401f385/ocaml/xapi/xapi_sr.ml#L209>
-=Tobias
Post by Vinícius Ferrão
Hello guys,
Answering about all the questions and issues. For those who aren't understanding the network topology, I've made a drawing of the solution. The image is here: https://www.dropbox.com/s/6zwerk6igcyqou3/UFRJ-IQ-VirtualPool.jpg?dl=0 <https://www.dropbox.com/s/6zwerk6igcyqou3/UFRJ-IQ-VirtualPool.jpg?dl=0>
192.168.10.1
192.168.11.1
192.168.20.1
192.168.21.1
The convention to this numbering scheme was: 192.168.{XenServerNumber}{InterfaceNumber}.{Storage(1)/Host(2)}.
192.168.10.2 (XenServer #1 - Interface #0)
192.168.11.2 (XenServer #1 - Interface #1)
192.168.20.2 (XenServer #2 - Interface #0)
192.168.21.2 (XenServer #2 - Interface #1)
Hope this completely clarifies the architecture/topology of the iSCSI network.
Restarting both hosts on the same time. [OK]
Restarting the second host. [OK]
Restarting the pool master. [OK]
2. HA is working too. It fences only the damaged host. I even tried a VM Failover. HA happened without any problem and the lost VM has been restarted on other server. To do the HA test, I've power cycled the host, simulated a hardware crash scenario. A cool picture: https://www.dropbox.com/s/g7h7rljk8vh97rg/Screenshot%202015-02-18%2015.45.41.png?dl=0 <https://www.dropbox.com/s/g7h7rljk8vh97rg/Screenshot%202015-02-18%2015.45.41.png?dl=0>
3. I haven't understood this test. You are asking to create a new LUN and add a new Shared Repository? It's not clear what you're saying with "independent SR".
4. Storage XenMotion works too. I've moved a running VM from local storage (on the Host #2) to the iSCSI pool without any problem. The reverse process works too.
5. I will test this tomorrow, when I will got physical access to the servers (carnival holiday here in BR). But about this specific test, I can't see any problem with plain Xen Motion, since the data is copied over the management network and not the iSCSI network. But I can see a good point when dealing with Storage XenMotion.
Finally about the changes on XenCenter or XenServer/XAPI itself. One thing to prove this point is to create the SR via the CLI. If I would be able to do this, the problem should definitely be on XenCenter. As I said before, I've created a broken Software iSCSI SR via XenCenter and them mapped the iSCSI interfaces, enabled multipath and created the PBD over the CLI.
The iSCSI and Multipath should be done in the physical host. But the PBD creation can be done without any problem on the Pool Master.
device-config: name-label= shared= type=
In this case sm-config and device-config are the problematic ones. Assuming that the host-uuid is always the pool master.
Thanks in advance,
Vinícius.
Post by Tim Mackey
I’ll add in a few more tests
.
1. Host restart. Does the normal PBD plug mechanism work with such a configuration
2. HA. While HA with two hosts isn’t a supported configuration, in this model it might work properly. If you remove the network cables for a single host, does it fence, or do both hosts fence?
3. Multiple SRs. If you have multiple LUNs on the FreeNAS exported and then used as independent SRs, does everything work as it should?
4. Storage XenMotion. With a vdi on one SR presented with the FreeNAS, copy it to any another storage option
5. Motion during path failure. With one path down on a given host, does XenMotion and Storage XenMotion still function correctly?
For those who are questioning the value of such a configuration, this is in essence a “cluster in a box” solution. These configurations were very popular 10-15 years ago as highly available compute infrastructure was quite expensive back then. Today we can accomplish the same things we did back then using virtualization, but “cluster in a box” can always be valuable for those with limited IT skills or for smaller organizations wanting least cost with fewest points of failure.
-tim
Sent: Monday, February 16, 2015 7:24 PM
To: Tobias Kreidl
Subject: Re: [xs-devel] Switchless shared iSCSI Network with /30 links
Tobias,
I've done more tests about the SR creation over XenCenter.
All the process definitely occurs only on the pool master. Even if the specified networks are not part of the networks of the host. The connection is tried on the default route.
192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1
The target IQN is discovered without any problem, since the iSCSI Storage reports the LUNs on any interface.
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 192.168.10.2:55870 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52654 192.168.11.1:3260 TIME_WAIT
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.11.2:52656 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
******************
tcp 0 1 10.7.0.101:56481 192.168.21.1:3260 SYN_SENT
******************
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
******************
tcp 0 1 10.7.0.101:52787 192.168.30.1:3260 SYN_SENT
******************
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
If you're wondering the 192.168.30.1 is reported because in the future we will add another XS host to the pool. The point here is that XenCenter should not lookup for this network since it isn't specified during the Software iSCSI creation. On my understanding it should only lookup for the specified addresses.
tcp 0 48 10.7.0.102:22 192.168.172.1:58046 ESTABLISHED
udp 0 0 192.168.21.2:123 0.0.0.0:*
udp 0 0 192.168.20.2:123 0.0.0.0:*
As you said, perhaps propagating the initial connection through the host will solve the problem without modifying the XenCenter interface. A more sophisticated approach would be query the network-list over XAPI, retrieve the PIFs associated to those networks and them just check the IP addresses on the SR creation request and match them with the equivalent PIFs.
uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa
name-label ( RW): iSCSI #0
PIF-uuids (SRO): 3baa8436-feca-cd04-3173-5002d9ffc66b; 362d63cb-647a-465f-6b81-1b59cd3b1f5d
MTU ( RW): 9000
bridge ( RO): xenbr2
other-config (MRW): automatic: true
default-locking-mode ( RW): unlocked
uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5c:64
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
host-name-label ( RO): xenserver2.local.iq.ufrj.br
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.20.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5a:1c
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c
host-name-label ( RO): xenserver1.local.iq.ufrj.br
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.10.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
So as you can see, we have the IP addresses and the network mask. Which is sufficient to an efficient and precise algorithm of SR creation.
I thinks this clarifies everything we discussed until now. A patch on XenCenter appears to be really viable.
Many thanks,
Vinicius.
Yes, correct. The MD36xx looks like two separate controller units, each acting as a separate device. For a standalone server or some storage devices, that's right that you need a separate subnet for each connection.
Because a single connection isn't seeing the broadcast from the other connectors at the time of the initial connection from one host, the pool is unaware of the others. The "repair" operation causes this to be conducted from each host, and as I guessed, then fixes the SR connectivity for the entire pool. I am speculating that the request for a new connection of this type via XenCenter could perhaps be solved by propagating that initial connection process to the rest of the hosts in the pool and then rescan the SR.
-=Tobias
Hi Tobias,
As for I know, the Dell MD36xx is a dual controller based storage. So it's basically "two computers". Considering this with the added knowledge that I've about TCP networking there's a golden rule that says: only one IP address on a given subnet on a single computer. So I can't use the same network on my case. That's why I use point-to-point links too.
Since I've only one machine acting as a storage server (It's a Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do need four separate networks. FreeNAS is FreeBSD based, and BSD enforces this rule by default. I can't have two IP addresses of the same subnet on same machine, even if I manually configure it.
If this wasn't bad TCP network practice, I would be able to configure the SR via XenCenter, since it will "find" the same network on the others hosts without problem.
About the XenCenter, it really appears to be only a missing function on the software. If I was a good programmer I would try to implement this. But this is beyond my knowledge... :(
Thanks once again,
Vinícius.
No reason to necessarily use four separate subnets. You could have 192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and 2.2 on the other controller (on, for example, an MD36xx that is standard). If however it is something like an MD32xx which recommends four separate subets, then of course, use four. One should follow the vendor's specifications.
I will respond in more detail later when I get a chance, Vinícius, but after a quick read, I agree with your findings from your other email. It would appear that the initial connection is indeed evidently a missing function in XenCenter and perhaps all that is needed to make this work "out of the box". As to the wildcard discovery, it's generally preferred as it allows the host to figure out the path on its own. The only reason I could see not using a wildcard discovery would be if there were any chance for overlaps and ambiguous connections.
Regards,
--Tobias
Hello Dave,
+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+
192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30
192.168.10.2/30
192.168.11.2/30
192.168.20.2/30
192.168.21.2/30
That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0 <https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0>
Cheers,
Vinícius.
Hi,
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked
 Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0 <https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0>
I’m really excited because it worked. So let’s me explain what I’ve did.
First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I’d like to understand your configuration better— have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?
Thanks,
Dave
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0 <https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0>
I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch <http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch>
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/ <https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/>
It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Tobias J Kreidl
2015-02-21 04:13:57 UTC
Permalink
IMO. this would take modifying the XenCenter code to trigger running the SrRepairAction routine if it sees multiple, comma-separated entries in the target input field to make a new iSCSi conenction. That would be an ugly way to fix things, but might at least work.

If you try this and get one host conencted via XenCEnter and then do an SR Repair operation via XenCenter does that still work to get all the hosts conencted?

________________________________

From: xs-devel-***@lists.xenserver.org [xs-devel-***@lists.xenserver.org] on behalf of Vinícius Ferrão [***@ferrao.eti.br]
Sent: Friday, February 20, 2015 8:49 PM
To: xs-***@lists.xenserver.org
Subject: Re: [xs-devel] Switchless shared iSCSI Network with /30 links

Any ideia on how to test this? I tried to create the pool for the first time using the string with all the six IP addresses, but XenCenter just fails without and error message, just an single X appears. The only thing I can provide at this moment is the /var/log/xensource.log; /var/log/audit.log; /var/log/messages and /var/log/SMlog file during the IQN/LUN phase, it's on the end of this message. The /var/log/SMlog appears to be the most valuable one.

I'm really dedicated on helping solving this issue, since I'm delaying the implementation of this farm just to keep the test scenario live!

Thanks,

PS: The logs...

*****/var/log/xensource.log*****
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|Async.SR.create R:b933cd75031b|dispatcher] spawning a new thread to handle the current task (trackid=8c8347a3f363fd40a9bd41eb1705050f)
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|audit] SR.create: name label = '__gui__'
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] SR.create name_label=__gui__ sm_config=[ ]
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|Async.SR.create R:b933cd75031b|storage_access] SR d403d0fc-4a14-7f99-fe92-dfc6301ed764 will be implemented by /services/SM/lvmoiscsi in VM OpaqueRef:47f2c383-de3e-730e-9836-d652d5eae646
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|mux] register SR d403d0fc-4a14-7f99-fe92-dfc6301ed764 (currently-registered = [ f31651b7-4219-dfe5-d871-4748c05e900c, ee47042b-76bf-5f62-2879-0457db8e892d, 12945887-3593-6c24-0b77-2efd62949415, d403d0fc-4a14-7f99-fe92-dfc6301ed764, 551b026c-6b94-5e08-ae67-3c76c15b8b44 ])
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|dummytaskhelper] task SR.create D:ebc6f0ef8713 created by task R:b933cd75031b
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|sm] SM lvmoiscsi sr_create sr=OpaqueRef:383eb943-faa8-1946-f169-6626b1996fe7 size=0
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Session.create trackid=70acc8c1d9618c45c2f07860af213730 pool=false uname= originator= is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|mscgen] xapi=>xapi [label="(XML)"];
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14339 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:7e7d99be1ccd created by task D:208b3f98f095
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|mscgen] smapiv2=>smapiv1 [label="sr_create"];
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:e234520429b7|api_effect] task.add_to_other_config
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|xenops] Event on VM 6dd2a08f-43a3-4f0c-b6c4-7e040829dc6a; resident_here = true
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|mscgen] xapi=>xenops [label="VM.exists"];
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|xenops] kernel is not in the whitelist: ignoring
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|mscgen] xapi=>xapi [label="(XML)"];
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14343 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:event.from D:ff5a707a4074 created by task D:4b65ef16de2c
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:7d79581bbf17|api_effect] task.add_to_other_config
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14344 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:27e54f4f8e11 created by task D:ebc6f0ef8713
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14345 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:fac3f1373b6e created by task D:ebc6f0ef8713
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14346 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:1713380916eb created by task D:ebc6f0ef8713
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:538ce8692691|api_effect] task.add_to_other_config
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Raised at sm_exec.ml:212.7-69 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Session.destroy trackid=70acc8c1d9618c45c2f07860af213730
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|backtrace] Raised at pervasiveext.ml:26.22-25 -> server_helpers.ml:76.11-23
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|dispatcher] Server_helpers.exec exception_handler: Got exception SR_BACKEND_FAILURE_96: [ ; The request is missing or has an incorrect target IQN parameter; <?xml version="1.0" ?> <iscsi-target-iqns> <TGT> <Index> 0 </Index> <IPAddress> 192.168.10.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 1 </Index> <IPAddress> 192.168.11.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 2 </Index> <IPAddress> 192.168.20.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 3 </Index> <IPAddress> 192.168.21.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 4 </Index> <IPAddress> 192.168.30.1 </IPAddress>
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|dispatcher] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|backtrace] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|backtrace] Raised at storage_access.ml:161.15-44 -> server_helpers.ml:76.11-23
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|dispatcher] Server_helpers.exec exception_handler: Got exception INTERNAL_ERROR: [ Storage_interface.Backend_error(_) ]
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|dispatcher] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|backtrace] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] Raised at pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [error|xenserver1|14336|Async.SR.create R:b933cd75031b|storage_access] Re-raising as SR_BACKEND_FAILURE_96 [ ; The request is missing or has an incorrect target IQN parameter; <?xml version="1.0" ?> <iscsi-target-iqns> <TGT> <Index> 0 </Index> <IPAddress> 192.168.10.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 1 </Index> <IPAddress> 192.168.11.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 2 </Index> <IPAddress> 192.168.20.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 3 </Index> <IPAddress> 192.168.21.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 4 </Index> <IPAddress> 192.168.30.1 </IPAddress> <TargetIQN>iqn.2015-01.b
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|backtrace] Raised at xapi_sr.ml:225.9-10 -> server.ml:21898.82-267 -> rbac.ml:229.16-23
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|backtrace] Raised at rbac.ml:238.10-15 -> server_helpers.ml:79.11-41
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|dispatcher] Server_helpers.exec exception_handler: Got exception SR_BACKEND_FAILURE_96: [ ; The request is missing or has an incorrect target IQN parameter; <?xml version="1.0" ?> <iscsi-target-iqns> <TGT> <Index> 0 </Index> <IPAddress> 192.168.10.1 </IPAddress> <TargetIQN>iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 1 </Index> <IPAddress> 192.168.11.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 2 </Index> <IPAddress>192.168.20.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 3 </Index> <IPAddress> 192.168.21.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 4 </Index> <IPAddress> 192.168.30.1 </IPAdd
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|dispatcher] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|backtrace] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336||xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336||xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|Async.SR.create R:202cf9100203|dispatcher] spawning a new thread to handle the current task (trackid=8c8347a3f363fd40a9bd41eb1705050f)
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|audit] SR.create: name label = '__gui__'
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] SR.create name_label=__gui__ sm_config=[ ]
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|Async.SR.create R:202cf9100203|storage_access] SR 207535ee-c14c-7b48-3869-96318c1c7877 will be implemented by /services/SM/lvmoiscsi in VM OpaqueRef:47f2c383-de3e-730e-9836-d652d5eae646
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|mux] register SR 207535ee-c14c-7b48-3869-96318c1c7877 (currently-registered = [ f31651b7-4219-dfe5-d871-4748c05e900c, ee47042b-76bf-5f62-2879-0457db8e892d, 207535ee-c14c-7b48-3869-96318c1c7877, 12945887-3593-6c24-0b77-2efd62949415, d403d0fc-4a14-7f99-fe92-dfc6301ed764, 551b026c-6b94-5e08-ae67-3c76c15b8b44 ])
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|dummytaskhelper] task SR.create D:7e6802e5fed5 created by task R:202cf9100203
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|sm] SM lvmoiscsi sr_create sr=OpaqueRef:5591eb47-1a35-9321-6aa4-bd8beb748cbf size=0
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Session.create trackid=836c7784c6fd5a42124fb4bd64aa2cd5 pool=false uname= originator= is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|mscgen] xapi=>xapi [label="(XML)"];
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14357 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:c14a1f2ccf30 created by task D:4a4af7d945d9
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|mscgen] smapiv2=>smapiv1 [label="sr_create"];
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:6f63db465b85|api_effect] task.add_to_other_config
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|xenops] Event on VM 6dd2a08f-43a3-4f0c-b6c4-7e040829dc6a; resident_here = true
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|mscgen] xapi=>xenops [label="VM.exists"];
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|xenops] kernel is not in the whitelist: ignoring
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|mscgen] xapi=>xapi [label="(XML)"];
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14359 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:event.from D:dd4a82303df3 created by task D:4b65ef16de2c
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:454990b8d01f|api_effect] task.add_to_other_config
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14360 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:513472992344 created by task D:7e6802e5fed5
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14361 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:4cb6c60153d2 created by task D:7e6802e5fed5
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14362 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:901d21392528 created by task D:7e6802e5fed5
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:f7f97c5c1dec|api_effect] task.add_to_other_config
Feb 21 01:41:17 xenserver1 xapi: [debug|xenserver1|14365 INET 0.0.0.0:80|Get RRD updates. D:8f822f0bc4c4|xapi] hand_over_connection GET /rrd_updates to /var/xapi/xcp-rrdd.forwarded
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Raised at forkhelpers.ml:187.30-76 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Raised at sm_exec.ml:190.10-100 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Raised at pervasiveext.ml:26.22-25 -> sm_exec.ml:173.23-1023 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [ info|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Session.destroy trackid=836c7784c6fd5a42124fb4bd64aa2cd5
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|backtrace] Raised at pervasiveext.ml:26.22-25 -> server_helpers.ml:76.11-23
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|dispatcher] Server_helpers.exec exception_handler: Got exception SR_BACKEND_FAILURE: [ non-zero exit; ; Traceback (most recent call last): File "/opt/xensource/sm/LVMoISCSISR", line 582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File "/opt/xensource/sm/SRCommand.py", line 343, in run sr = driver(cmd, cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line 142, in __init__ self.load(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line 156, in load self.iscsi = self.iscsiSRs[0] IndexError: list index out of range ]
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|dispatcher] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|backtrace] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|backtrace] Raised at storage_access.ml:161.15-44 -> server_helpers.ml:76.11-23
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|dispatcher] Server_helpers.exec exception_handler: Got exception INTERNAL_ERROR: [ Storage_interface.Backend_error(_) ]
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|dispatcher] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|backtrace] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] Raised at pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [error|xenserver1|14352|Async.SR.create R:202cf9100203|storage_access] Re-raising as SR_BACKEND_FAILURE [ non-zero exit; ; Traceback (most recent call last): File "/opt/xensource/sm/LVMoISCSISR", line 582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File "/opt/xensource/sm/SRCommand.py", line 343, in run sr = driver(cmd, cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line 142, in __init__ self.load(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line 156, in load self.iscsi = self.iscsiSRs[0] IndexError: list index out of range ]
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|backtrace] Raised at xapi_sr.ml:225.9-10 -> server.ml:21898.82-267 -> rbac.ml:229.16-23
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|backtrace] Raised at rbac.ml:238.10-15 -> server_helpers.ml:79.11-41
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|dispatcher] Server_helpers.exec exception_handler: Got exception SR_BACKEND_FAILURE: [ non-zero exit; ; Traceback (most recent call last): File "/opt/xensource/sm/LVMoISCSISR", line 582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File "/opt/xensource/sm/SRCommand.py", line 343, in run sr = driver(cmd, cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line 142, in __init__ self.load(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line 156, in load self.iscsi = self.iscsiSRs[0] IndexError: list index out of range ]
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|dispatcher] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|backtrace] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352||xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352||xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0 monitor|main|rrdd_stats] system stats: MemTotal: 3959600 KiB; MemFree: 3262140 KiB; Buffered: 33832 KiB; Cached: 291048 KiB; SwapTotal: 524284 KiB; SwapFree: 524284 KiB
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0 monitor|main|rrdd_stats] Clock drift: -0
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0 monitor|main|rrdd_stats] xcp-rrdd stats (n = 1): size: 155052 KiB; rss: 8368 KiB; data: 71620 KiB; stack: 136 KiB
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0 monitor|main|rrdd_stats] xapi stats (n = 2): size: 604328 KiB; rss: 101844 KiB; data: 451340 KiB; stack: 272 KiB
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0 monitor|main|rrdd_stats] xenopsd stats (n = 1): size: 278248 KiB; rss: 8308 KiB; data: 193692 KiB; stack: 136 KiB




***** /var/log/audit.log *****
Feb 21 01:41:01 xenserver1 xapi: [20150221T03:41:01.707Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:ee0292d7107f|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' '577acdd7-2b8c-b177-bf45-f8422c4ed7ca' 'OpaqueRef:b933cd75-031b-6ff0-5a0a-2341507af648')))
Feb 21 01:41:01 xenserver1 xapi: [20150221T03:41:01.747Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:ef6d521270fb|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' '577acdd7-2b8c-b177-bf45-f8422c4ed7ca' 'OpaqueRef:b933cd75-031b-6ff0-5a0a-2341507af648') ('value' '' 'e1aaff51-0da2-2745-cf25-ab034aea8a25' 'OpaqueRef:85f25443-719e-105d-35aa-a1c1043fdd8e')))
Feb 21 01:41:01 xenserver1 xapi: [20150221T03:41:01.787Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:43071305947a|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' '577acdd7-2b8c-b177-bf45-f8422c4ed7ca' 'OpaqueRef:b933cd75-031b-6ff0-5a0a-2341507af648')))
Feb 21 01:41:01 xenserver1 xapi: [20150221T03:41:01.885Z|audit|xenserver1|14336|Async.SR.create R:b933cd75031b|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:SR_BACKEND_FAILURE_96: [ ; The request is missing or has an incorrect target IQN parameter; <?xml version=\"1.0\" ?> <iscsi-target-iqns> <TGT> <Index> 0 </Index> <IPAddress> 192.168.10.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 1 </Index> <IPAddress> 192.168.11.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 2 </Index> <IPAddress> 192.168.20.1 </IPAddress> <TargetIQN>iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 3 </Index> <IPAddress> 192.168.21.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index>
Feb 21 01:41:02 xenserver1 xapi: [20150221T03:41:02.883Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.destroy D:5804968b2ed6|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.destroy' (('self' 'Async.SR.create' '577acdd7-2b8c-b177-bf45-f8422c4ed7ca' 'OpaqueRef:b933cd75-031b-6ff0-5a0a-2341507af648')))
Feb 21 01:41:06 xenserver1 xapi: [20150221T03:41:06.598Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:a09ea039fcd0|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' 'a0d8e973-9995-fb15-532e-f36ac5184c28' 'OpaqueRef:202cf910-0203-a943-9054-8d0e25a2b4a7')))
Feb 21 01:41:06 xenserver1 xapi: [20150221T03:41:06.638Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:b4dd3f0d946a|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' 'a0d8e973-9995-fb15-532e-f36ac5184c28' 'OpaqueRef:202cf910-0203-a943-9054-8d0e25a2b4a7') ('value' '' 'e1aaff51-0da2-2745-cf25-ab034aea8a25' 'OpaqueRef:85f25443-719e-105d-35aa-a1c1043fdd8e')))
Feb 21 01:41:06 xenserver1 xapi: [20150221T03:41:06.679Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:9b0a16a91401|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' 'a0d8e973-9995-fb15-532e-f36ac5184c28' 'OpaqueRef:202cf910-0203-a943-9054-8d0e25a2b4a7')))
Feb 21 01:41:17 xenserver1 xapi: [20150221T03:41:18.001Z|audit|xenserver1|14365 INET 0.0.0.0:80|handler:http/rrd_updates D:3d1bf1a8b00c|audit] ('trackid=bf92de30818eb264b5673ffe734ad369' 'LOCAL_SUPERUSER' '' 'ALLOWED' 'OK' 'HTTP' 'http/rrd_updates' ())
Feb 21 01:41:35 xenserver1 xapi: [20150221T03:41:35.155Z|audit|xenserver1|14352|Async.SR.create R:202cf9100203|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:SR_BACKEND_FAILURE: [ non-zero exit; ; Traceback (most recent call last): File \"/opt/xensource/sm/LVMoISCSISR\", line 582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File \"/opt/xensource/sm/SRCommand.py\", line 343, in run sr = driver(cmd, cmd.sr_uuid) File \"/opt/xensource/sm/SR.py\", line 142, in __init__ self.load(sr_uuid) File \"/opt/xensource/sm/LVMoISCSISR\", line 156, in load self.iscsi = self.iscsiSRs[0] IndexError: list index out of range ]' 'API' 'SR.create' (('host' 'xenserver1.local.iq.ufrj.br<http://xenserver1.local.iq.ufrj.br>' 'c8927969-b7bf-4a0f-9725-035550381d9c' 'OpaqueRef:03239cf1-76cf-9c86-b927-56ea27cb6b94') ('name_label' '__gui__' '' '') ('name_description' 'SHOULD NEVER BE CREATED' '' '')))
Feb 21 01:41:35 xenserver1 xapi: [20150221T03:41:35.707Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.destroy D:7ff3ff7db08a|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.destroy' (('self' 'Async.SR.create' 'a0d8e973-9995-fb15-532e-f36ac5184c28' 'OpaqueRef:202cf910-0203-a943-9054-8d0e25a2b4a7')))





***** /var/log/messages *****
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|Async.SR.create R:b933cd75031b|dispatcher] spawning a new thread to handle the current task (trackid=8c8347a3f363fd40a9bd41eb1705050f)
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|Async.SR.create R:b933cd75031b|storage_access] SR d403d0fc-4a14-7f99-fe92-dfc6301ed764 will be implemented by /services/SM/lvmoiscsi in VM OpaqueRef:47f2c383-de3e-730e-9836-d652d5eae646
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Session.create trackid=70acc8c1d9618c45c2f07860af213730 pool=false uname= originator= is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Session.destroy trackid=70acc8c1d9618c45c2f07860af213730
Feb 21 01:41:01 xenserver1 xapi: [error|xenserver1|14336|Async.SR.create R:b933cd75031b|storage_access] Re-raising as SR_BACKEND_FAILURE_96 [ ; The request is missing or has an incorrect target IQN parameter; <?xml version="1.0" ?> <iscsi-target-iqns> <TGT> <Index> 0 </Index> <IPAddress> 192.168.10.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 1 </Index> <IPAddress> 192.168.11.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 2 </Index> <IPAddress> 192.168.20.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 3 </Index> <IPAddress> 192.168.21.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 4 </Index> <IPAddress> 192.168.30.1 </IPAddress> <TargetIQN>iqn.2015-01.b
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|Async.SR.create R:202cf9100203|dispatcher] spawning a new thread to handle the current task (trackid=8c8347a3f363fd40a9bd41eb1705050f)
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|Async.SR.create R:202cf9100203|storage_access] SR 207535ee-c14c-7b48-3869-96318c1c7877 will be implemented by /services/SM/lvmoiscsi in VM OpaqueRef:47f2c383-de3e-730e-9836-d652d5eae646
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Session.create trackid=836c7784c6fd5a42124fb4bd64aa2cd5 pool=false uname= originator= is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05155|bond|INFO|bond bond0: shift 21kB of load (with hash 35) from eth1 to eth0 (now carrying 233kB and 94kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05156|bond|INFO|bond bond0: shift 224kB of load (with hash 39) from eth1 to eth0 (now carrying 9kB and 318kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05157|bond|INFO|bond bond0: shift 0kB of load (with hash 9) from eth0 to eth1 (now carrying 318kB and 9kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05158|bond|INFO|bond bond0: shift 1kB of load (with hash 12) from eth0 to eth1 (now carrying 317kB and 10kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05159|bond|INFO|bond bond0: shift 1kB of load (with hash 14) from eth0 to eth1 (now carrying 316kB and 11kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05160|bond|INFO|bond bond0: shift 0kB of load (with hash 20) from eth0 to eth1 (now carrying 315kB and 11kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05161|bond|INFO|bond bond0: shift 0kB of load (with hash 21) from eth0 to eth1 (now carrying 315kB and 12kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05162|bond|INFO|bond bond0: shift 0kB of load (with hash 25) from eth0 to eth1 (now carrying 315kB and 12kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05163|bond|INFO|bond bond0: shift 15kB of load (with hash 46) from eth0 to eth1 (now carrying 300kB and 27kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05164|bond|INFO|bond bond0: shift 2kB of load (with hash 70) from eth0 to eth1 (now carrying 297kB and 30kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05165|bond|INFO|bond bond0: shift 1kB of load (with hash 84) from eth0 to eth1 (now carrying 295kB and 32kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05166|bond|INFO|bond bond0: shift 0kB of load (with hash 94) from eth0 to eth1 (now carrying 295kB and 32kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05167|bond|INFO|bond bond0: shift 2kB of load (with hash 112) from eth0 to eth1 (now carrying 293kB and 34kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05168|bond|INFO|bond bond0: shift 28kB of load (with hash 118) from eth0 to eth1 (now carrying 265kB and 62kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05169|bond|INFO|bond bond0: shift 5kB of load (with hash 160) from eth0 to eth1 (now carrying 259kB and 68kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05170|bond|INFO|bond bond0: shift 5kB of load (with hash 171) from eth0 to eth1 (now carrying 254kB and 73kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05171|bond|INFO|bond bond0: shift 3kB of load (with hash 206) from eth0 to eth1 (now carrying 251kB and 76kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05172|bond|INFO|bond bond0: shift 2kB of load (with hash 254) from eth0 to eth1 (now carrying 249kB and 78kB load, respectively)
Feb 21 01:41:28 xenserver1 ovs-vswitchd: ovs|05173|ofproto|INFO|xapi0: 29 flow_mods in the last 59 s (29 adds)
Feb 21 01:41:35 xenserver1 fe: 2498 (/opt/xensource/sm/LVMoISCSISR <methodCall><methodName>sr_create</methodName><...) exited with code 1
Feb 21 01:41:35 xenserver1 xapi: [ info|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Session.destroy trackid=836c7784c6fd5a42124fb4bd64aa2cd5
Feb 21 01:41:35 xenserver1 xapi: [error|xenserver1|14352|Async.SR.create R:202cf9100203|storage_access] Re-raising as SR_BACKEND_FAILURE [ non-zero exit; ; Traceback (most recent call last): File "/opt/xensource/sm/LVMoISCSISR", line 582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File "/opt/xensource/sm/SRCommand.py", line 343, in run sr = driver(cmd, cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line 142, in __init__ self.load(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line 156, in load self.iscsi = self.iscsiSRs[0] IndexError: list index out of range ]
Feb 21 01:41:56 xenserver1 xenstored: D3.14 rm attr/eth0
Feb 21 01:41:56 xenserver1 xenstored: D3 write attr/eth0/ip 10.7.0.243
Feb 21 01:41:56 xenserver1 xenstored: D3 write attr/eth0/ipv6/0/addr fe80::484b:acff:fe91:5b19



***** /var/log/SMlog *****
[***@xenserver1 log]# grep "Feb 21 01:41" /var/log/SMlog
Feb 21 01:41:01 xenserver1 SM: [2444] _testHost: Testing host/port: 192.168.10.1,3260
Feb 21 01:41:01 xenserver1 SM: [2444] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] Raising exception [96, The request is missing or has an incorrect target IQN parameter]
Feb 21 01:41:06 xenserver1 SM: [2498] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] Raising exception [202, General backend error [opterr=rc: 21, stdout: , stderr: iscsiadm: No active sessions.
Feb 21 01:41:06 xenserver1 SM: [2498] ]]
Feb 21 01:41:06 xenserver1 SM: [2498] ['iscsiadm', '-m', 'session'] failed with (u'General backend error [opterr=rc: 21, stdout: , stderr: iscsiadm: No active sessions.\n]',)
Feb 21 01:41:06 xenserver1 SM: [2498] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] ['ls', '/sys/class/scsi_host', '-1', '--color=never']
Feb 21 01:41:06 xenserver1 SM: [2498] pread SUCCESS
Feb 21 01:41:06 xenserver1 SM: [2498] []
Feb 21 01:41:06 xenserver1 SM: [2498] PATHDICT: key 192.168.10.1:3260: {'path': '/dev/iscsi/*/192.168.10.1:3260', 'ipaddr': '192.168.10.1', 'port': 3260L}
Feb 21 01:41:06 xenserver1 SM: [2498] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] Discovery for IP 192.168.10.1 returned [('192.168.10.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.11.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.20.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.21.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.30.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.31.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1')]
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.10.1,3260
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.11.1,3260
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.20.1,3260
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.20.1) - (113, 'No route to host')
Feb 21 01:41:06 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:06 xenserver1 SM: [2498] Target Not reachable: (192.168.20.1:3260)
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.21.1,3260
Feb 21 01:41:07 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.21.1) - (113, 'No route to host')
Feb 21 01:41:07 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:07 xenserver1 SM: [2498] Target Not reachable: (192.168.21.1:3260)
Feb 21 01:41:07 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.30.1,3260
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.30.1) - timed out
Feb 21 01:41:09 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:09 xenserver1 SM: [2498] Target Not reachable: (192.168.30.1:3260)
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.31.1,3260
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.31.1) - (113, 'No route to host')
Feb 21 01:41:09 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:09 xenserver1 SM: [2498] Target Not reachable: (192.168.31.1:3260)
Feb 21 01:41:09 xenserver1 SM: [2498] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:09 xenserver1 SM: [2498] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:09 xenserver1 SM: [2498] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:09 xenserver1 SM: [2498] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:09 xenserver1 SM: [2498] Discovery for IP 192.168.11.1 returned [('192.168.10.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.11.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.20.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.21.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.30.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.31.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1')]
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.10.1,3260
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.11.1,3260
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.20.1,3260
Feb 21 01:41:11 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.20.1) - timed out
Feb 21 01:41:11 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:11 xenserver1 SM: [2498] Target Not reachable: (192.168.20.1:3260)
Feb 21 01:41:11 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.21.1,3260
Feb 21 01:41:11 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.21.1) - (113, 'No route to host')
Feb 21 01:41:11 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:11 xenserver1 SM: [2498] Target Not reachable: (192.168.21.1:3260)
Feb 21 01:41:11 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.30.1,3260
Feb 21 01:41:12 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.30.1) - (113, 'No route to host')
Feb 21 01:41:12 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:12 xenserver1 SM: [2498] Target Not reachable: (192.168.30.1:3260)
Feb 21 01:41:12 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.31.1,3260
Feb 21 01:41:14 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.31.1) - timed out
Feb 21 01:41:14 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:14 xenserver1 SM: [2498] Target Not reachable: (192.168.31.1:3260)
Feb 21 01:41:15 xenserver1 SM: [2498] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:15 xenserver1 SM: [2498] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:35 xenserver1 SM: [2498] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:35 xenserver1 SM: [2498] Raising exception [202, General backend error [opterr=rc: 21, stdout: , stderr: iscsiadm: cannot make connection to 192.168.20.1: No route to host
Feb 21 01:41:35 xenserver1 SM: [2498] iscsiadm: cannot make connection to 192.168.20.1: No route to host
Feb 21 01:41:35 xenserver1 last message repeated 3 times
Feb 21 01:41:35 xenserver1 SM: [2498] iscsiadm: connect to 192.168.20.1 timed out
Feb 21 01:41:35 xenserver1 SM: [2498] iscsiadm: connection login retries (reopen_max) 5 exceeded
Feb 21 01:41:35 xenserver1 SM: [2498] iscsiadm: No portals found
Feb 21 01:41:35 xenserver1 SM: [2498] ]]
Feb 21 01:41:35 xenserver1 SM: [2498] Raising exception [68, ISCSI login failed, verify CHAP credentials]
Feb 21 01:41:35 xenserver1 SM: [2498] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:35 xenserver1 SM: [2498] ***** LVHDoISCSISR.load: EXCEPTION SR.SROSError, ISCSI login failed, verify CHAP credentials
Feb 21 01:41:35 xenserver1 SM: [2498] File "/opt/xensource/sm/LVMoISCSISR", line 119, in load
Feb 21 01:41:35 xenserver1 SM: [2498] map = iscsilib.discovery(tgt_ip,iscsi.port,iscsi.chapuser,iscsi.chappassword,targetIQN=IQN)
Feb 21 01:41:35 xenserver1 SM: [2498] File "/opt/xensource/sm/iscsilib.py", line 156, in discovery
Feb 21 01:41:35 xenserver1 SM: [2498] raise xs_errors.XenError('ISCSILogin')
Feb 21 01:41:35 xenserver1 SM: [2498] File "/opt/xensource/sm/xs_errors.py", line 52, in __init__
Feb 21 01:41:35 xenserver1 SM: [2498] raise SR.SROSError(errorcode, errormessage)
Feb 21 01:41:35 xenserver1 SM: [2498]
Feb 21 01:41:54 xenserver1 updatempppathd: [7617] The garbage collection routine returned: 0


On Feb 20, 2015, at 9:14 PM, Tobias Kreidl <***@nau.edu<mailto:***@nau.edu>> wrote:

When I mentioned running the SrRepairAction procedure, I meant in conjunction with what is not happening in XenCenter as a follow-up to fix things.

So then, it would seem the piece remaining to resolve is what is missing in XenCenter that is not triggered. My guess is that it can perhaps only perform a connection using one IP address (or wildcarded address) and hence apparently doesn't understand what to do with anything involving a multiSession/multihome list?

It would be a useful diagnostic to see what is parsed out and processed if you pass in the whole multihome list of

192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260

to XenCenter and see what it ends up trying to do with it.

-=Tobias

On 2/20/2015 3:53 PM, Vinícius Ferrão wrote:

Guys, another important thing that I've observed.

All the iscsiadm and multipath commands are useless. I've attached the same storage just issuing a specific PBD command from the Pool Master. So this simplifies the problem a lot.

I just used this command:

xe pbd-create \
sr-uuid=f8d481b8-be10-922e-b875-91c4d73d341d \
host-uuid=44955863-f4a3-4770-bd7a-f4342dd7925a \
device-config:multiSession="192.168.30.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.31.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" \
device-config:target=192.168.30.1,192.168.31.1 \
device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 \
device-config:targetIQN=* \
device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 \
device-config:port=3260

The device-config attributes are sufficient to create all the connection.

Thanks,
Vinicius.



On Feb 20, 2015, at 8:37 PM, Tobias Kreidl <***@nau.edu><mailto:***@nau.edu> wrote:

It may be possible to at least temporarily get this work by forcing the SrRepairAction procedure to be run.
I didn't have time yet, but wanted to compare what it does in sequence vs. what is done by default in the xapi_sr
program to see what is different. As far as XenCenter is concerned, perhaps if multiple IP target addresses are entered in the dialog box (as opposed to a single one or a wildcard entry), then a different means could be triggered to ensure that each host does a proper PBD creation and plug-in for that new SR.

-=Tobias


On 2/20/2015 3:20 PM, Vinícius Ferrão wrote:


Dave,

I've added the "future" server to our pool. And happened exactly what you said. The configurations were cloned. But obviously it failed to clone the PDB's and the Network configuration for the iSCSI network.

IMHO there are two things that needs to be addressed:
1. The network configuration on added host should be "confirmed" to the user, and not done automatically, so the user can adapt is to specific needs.
2. After this the PDB creation should be done accordingly. If the network topology has changed by the user, so the PDBs must be created with this new topology.

In resume, I've three host at this moment on the pool. So I can do the HA tests once again, and I will report to the list the results. At this moment the only thing that failed is cloning the iSCSI network configurations and plugging the PDBs. Only the bounded NIC0+NIC1 was created on the added host.

Thanks in advance,
Vinícius.



On Feb 18, 2015, at 4:50 PM, Dave Scott <***@citrix.com><mailto:***@citrix.com> wrote:




On 18 Feb 2015, at 18:03, Tobias Kreidl <***@nau.edu><mailto:***@nau.edu> wrote:

Vinícius:

Regarding Point 3) I believe what is meant is that you should test having more than one SR that you are connecting to using the same IP addresses for the target array. You know it works fine to create one SR, so can you add additional ones the same way and have them all live together in harmony? :-)

As to the sr-create command, the host-uuid is an optional parameter and hence does not need to be named for the pool master or any other. The only required parameters are the name-label and the SR type. Have you tried that to see if the behavior is any different compared to when you do supply a host name?


I believe the underlying Xapi data model can express what you want, but there are 1 or 2 places where Xapi will helpfully create PBDs for you by cloning the master, which you just need to be aware of.

For example you should be able to use something like

$ xe sr-create .. host-uuid=<any host> shared=false

Setting "shared=false" initially should avoid Xapi 'helpfully' pre-creating identical PBDs for every host with the same configuration[1]. You should end up with an SR attached to a single host.

If you then set the SR to shared=true you should be able to create and plug the exact PBDs you want:

$ xe sr-param-set uuid= shared=true

$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...
$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...

[ as an aside, it's obviously not great that the PBDs you get depend on exactly when you declare the SR is shared. This has happened because the original design had all PBDs being distinct, and the notion of 'copying the master' was added relatively late ]

Whenever a new host is added to the pool, Xapi will clone the master's PBD for you- you would have to destroy it and re-create it. I suppose cloning the master's configuration is as good a guess as any in this situation :-)

Dave

[1] https://github.com/xapi-project/xen-api/blob/b6f28c52fd9726553968b410e6b8cced5401f385/ocaml/xapi/xapi_sr.ml#L209



-=Tobias

On 2/18/2015 10:47 AM, Vinícius Ferrão wrote:


Hello guys,

Answering about all the questions and issues. For those who aren't understanding the network topology, I've made a drawing of the solution. The image is here: https://www.dropbox.com/s/6zwerk6igcyqou3/UFRJ-IQ-VirtualPool.jpg?dl=0

In the drawing you can see the iSCSI Network made with point-to-point links. The IP's on the storage box are:
192.168.10.1
192.168.11.1
192.168.20.1
192.168.21.1

The convention to this numbering scheme was: 192.168.{XenServerNumber}{InterfaceNumber}.{Storage(1)/Host(2)}.

So on the host #1 I've two addresses and more two on host #2:
192.168.10.2 (XenServer #1 - Interface #0)
192.168.11.2 (XenServer #1 - Interface #1)
192.168.20.2 (XenServer #2 - Interface #0)
192.168.21.2 (XenServer #2 - Interface #1)

Hope this completely clarifies the architecture/topology of the iSCSI network.

Talking about the tests raised by Tim:

1. Already done this test. Tried three different tests:


Restarting both hosts on the same time. [OK]


Restarting the second host. [OK]


Restarting the pool master. [OK]

2. HA is working too. It fences only the damaged host. I even tried a VM Failover. HA happened without any problem and the lost VM has been restarted on other server. To do the HA test, I've power cycled the host, simulated a hardware crash scenario. A cool picture: https://www.dropbox.com/s/g7h7rljk8vh97rg/Screenshot%202015-02-18%2015.45.41.png?dl=0

3. I haven't understood this test. You are asking to create a new LUN and add a new Shared Repository? It's not clear what you're saying with "independent SR".

4. Storage XenMotion works too. I've moved a running VM from local storage (on the Host #2) to the iSCSI pool without any problem. The reverse process works too.

5. I will test this tomorrow, when I will got physical access to the servers (carnival holiday here in BR). But about this specific test, I can't see any problem with plain Xen Motion, since the data is copied over the management network and not the iSCSI network. But I can see a good point when dealing with Storage XenMotion.

Finally about the changes on XenCenter or XenServer/XAPI itself. One thing to prove this point is to create the SR via the CLI. If I would be able to do this, the problem should definitely be on XenCenter. As I said before, I've created a broken Software iSCSI SR via XenCenter and them mapped the iSCSI interfaces, enabled multipath and created the PBD over the CLI.

The iSCSI and Multipath should be done in the physical host. But the PBD creation can be done without any problem on the Pool Master.

To test this scenario I do require some help crafting the xe sr-create command, because I don't know exactly how this command is formed by XenCenter, and it requires some doubtful inputs:

[***@xenserver1 ~]# xe sr-create
content-type= host-uuid= physical-size= sm-config:
device-config: name-label= shared= type=

In this case sm-config and device-config are the problematic ones. Assuming that the host-uuid is always the pool master.

Thanks in advance,
Vinícius.



On Feb 18, 2015, at 2:36 PM, Tim Mackey <***@citrix.com><mailto:***@citrix.com> wrote:

I'll add in a few more tests....

1. Host restart. Does the normal PBD plug mechanism work with such a configuration
2. HA. While HA with two hosts isn't a supported configuration, in this model it might work properly. If you remove the network cables for a single host, does it fence, or do both hosts fence?
3. Multiple SRs. If you have multiple LUNs on the FreeNAS exported and then used as independent SRs, does everything work as it should?
4. Storage XenMotion. With a vdi on one SR presented with the FreeNAS, copy it to any another storage option
5. Motion during path failure. With one path down on a given host, does XenMotion and Storage XenMotion still function correctly?

For those who are questioning the value of such a configuration, this is in essence a "cluster in a box" solution. These configurations were very popular 10-15 years ago as highly available compute infrastructure was quite expensive back then. Today we can accomplish the same things we did back then using virtualization, but "cluster in a box" can always be valuable for those with limited IT skills or for smaller organizations wanting least cost with fewest points of failure.

-tim

From: xs-devel-***@lists.xenserver.org<mailto:xs-devel-***@lists.xenserver.org> [mailto:xs-devel-***@lists.xenserver.org] On Behalf Of Vinícius Ferrão
Sent: Monday, February 16, 2015 7:24 PM
To: Tobias Kreidl
Cc: xs-***@lists.xenserver.org<mailto:xs-***@lists.xenserver.org>
Subject: Re: [xs-devel] Switchless shared iSCSI Network with /30 links

Tobias,

I've done more tests about the SR creation over XenCenter.

All the process definitely occurs only on the pool master. Even if the specified networks are not part of the networks of the host. The connection is tried on the default route.

I requested a new Software iSCSI SR with the following address:
192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1

The target IQN is discovered without any problem, since the iSCSI Storage reports the LUNs on any interface.

But when trying to discover the LUN, the process only is done on the pool master. During the discovery process, I left a watch command on both XS to with "netstat -an | grep 192.168" and the results are:

On the pool master, there's a lot of things going on:
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 192.168.10.2:55870 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52654 192.168.11.1:3260 TIME_WAIT
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.11.2:52656 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*

During the discovery, look at the 192.168.21.1 address being searched through the default route:
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
******************
tcp 0 1 10.7.0.101:56481 192.168.21.1:3260 SYN_SENT
******************
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*

Even addresses that does not exist on the pool (since the IQN discovery process reports them):
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
******************
tcp 0 1 10.7.0.101:52787 192.168.30.1:3260 SYN_SENT
******************
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*

If you're wondering the 192.168.30.1 is reported because in the future we will add another XS host to the pool. The point here is that XenCenter should not lookup for this network since it isn't specified during the Software iSCSI creation. On my understanding it should only lookup for the specified addresses.

And to finish the tests, the second XS host remains equally during all the SR creation phase:
[***@xenserver2 ~]# netstat -an | grep -i 192.168
tcp 0 48 10.7.0.102:22 192.168.172.1:58046 ESTABLISHED
udp 0 0 192.168.21.2:123 0.0.0.0:*
udp 0 0 192.168.20.2:123 0.0.0.0:*

As you said, perhaps propagating the initial connection through the host will solve the problem without modifying the XenCenter interface. A more sophisticated approach would be query the network-list over XAPI, retrieve the PIFs associated to those networks and them just check the IP addresses on the SR creation request and match them with the equivalent PIFs.

For example here is the output of the xe network-param-list to retrieve the associated PIF's and the xe pif-param-lst with the UUID's of the retrieved PIFs:

[***@xenserver1 ~]# xe network-param-list uuid=4dc936d7-e806-c69a-a292-78e0ed2d5faa
uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa
name-label ( RW): iSCSI #0
name-description ( RW):
VIF-uuids (SRO):
PIF-uuids (SRO): 3baa8436-feca-cd04-3173-5002d9ffc66b; 362d63cb-647a-465f-6b81-1b59cd3b1f5d
MTU ( RW): 9000
bridge ( RO): xenbr2
other-config (MRW): automatic: true
blobs ( RO):
tags (SRW):
default-locking-mode ( RW): unlocked


[***@xenserver1 ~]# xe pif-param-list uuid=3baa8436-feca-cd04-3173-5002d9ffc66b
uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5c:64
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-master-of ( RO):
bond-slave-of ( RO): <not in database>
tunnel-access-PIF-of ( RO):
tunnel-transport-PIF-of ( RO):
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
host-name-label ( RO): xenserver2.local.iq.ufrj.br<http://xenserver2.local.iq.ufrj.br>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.20.2
netmask ( RO): 255.255.255.252
gateway ( RO):
IPv6-configuration-mode ( RO): None
IPv6 ( RO):
IPv6-gateway ( RO):
primary-address-type ( RO): IPv4
DNS ( RO):
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0


[***@xenserver1 ~]# xe pif-param-list uuid=362d63cb-647a-465f-6b81-1b59cd3b1f5d
uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5a:1c
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-master-of ( RO):
bond-slave-of ( RO): <not in database>
tunnel-access-PIF-of ( RO):
tunnel-transport-PIF-of ( RO):
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c
host-name-label ( RO): xenserver1.local.iq.ufrj.br<http://xenserver1.local.iq.ufrj.br>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.10.2
netmask ( RO): 255.255.255.252
gateway ( RO):
IPv6-configuration-mode ( RO): None
IPv6 ( RO):
IPv6-gateway ( RO):
primary-address-type ( RO): IPv4
DNS ( RO):
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0

So as you can see, we have the IP addresses and the network mask. Which is sufficient to an efficient and precise algorithm of SR creation.

I thinks this clarifies everything we discussed until now. A patch on XenCenter appears to be really viable.

Many thanks,
Vinicius.

On Feb 16, 2015, at 9:02 PM, Tobias Kreidl <***@nau.edu><mailto:***@nau.edu> wrote:

Yes, correct. The MD36xx looks like two separate controller units, each acting as a separate device. For a standalone server or some storage devices, that's right that you need a separate subnet for each connection.

Because a single connection isn't seeing the broadcast from the other connectors at the time of the initial connection from one host, the pool is unaware of the others. The "repair" operation causes this to be conducted from each host, and as I guessed, then fixes the SR connectivity for the entire pool. I am speculating that the request for a new connection of this type via XenCenter could perhaps be solved by propagating that initial connection process to the rest of the hosts in the pool and then rescan the SR.

-=Tobias

On 2/16/2015 2:20 PM, Vinícius Ferrão wrote:

Hi Tobias,

As for I know, the Dell MD36xx is a dual controller based storage. So it's basically "two computers". Considering this with the added knowledge that I've about TCP networking there's a golden rule that says: only one IP address on a given subnet on a single computer. So I can't use the same network on my case. That's why I use point-to-point links too.

Since I've only one machine acting as a storage server (It's a Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do need four separate networks. FreeNAS is FreeBSD based, and BSD enforces this rule by default. I can't have two IP addresses of the same subnet on same machine, even if I manually configure it.

If this wasn't bad TCP network practice, I would be able to configure the SR via XenCenter, since it will "find" the same network on the others hosts without problem.

About the XenCenter, it really appears to be only a missing function on the software. If I was a good programmer I would try to implement this. But this is beyond my knowledge... :(

Thanks once again,
Vinícius.


On Feb 16, 2015, at 6:53 PM, Tobias Kreidl <***@nau.edu><mailto:***@nau.edu> wrote:

No reason to necessarily use four separate subnets. You could have 192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and 2.2 on the other controller (on, for example, an MD36xx that is standard). If however it is something like an MD32xx which recommends four separate subets, then of course, use four. One should follow the vendor's specifications.

I will respond in more detail later when I get a chance, Vinícius, but after a quick read, I agree with your findings from your other email. It would appear that the initial connection is indeed evidently a missing function in XenCenter and perhaps all that is needed to make this work "out of the box". As to the wildcard discovery, it's generally preferred as it allows the host to figure out the path on its own. The only reason I could see not using a wildcard discovery would be if there were any chance for overlaps and ambiguous connections.

Regards,
--Tobias


On 2/16/2015 1:38 PM, Vinícius Ferrão wrote:

Hello Dave,

In total I have four paths. Two in each host. I've made a drawning using ASCII characters, hope this can clarify the architecture:

+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+

The Storage has four gigabit ethernet adapters with the following IP addresses:

192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30

On the first XenServer there are two interfaces with those IP's:

192.168.10.2/30
192.168.11.2/30

And on the second host the equivalent configuration:

192.168.20.2/30
192.168.21.2/30

That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.

To exemplify a little more heres a screenshot of the network configuration made on XenCenter:
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0

Cheers,
Vinícius.


On Feb 15, 2015, at 12:22 PM, Dave Scott <***@citrix.com><mailto:***@citrix.com> wrote:

Hi,


On 15 Feb 2015, at 05:44, Vinícius Ferrão <***@ferrao.eti.br><mailto:***@ferrao.eti.br> wrote:

Hello Tobias,
Thanks for the reply, even to discourage me :)

I really can't understand why it isn't a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I'm trying to do is achieve the same topology with XenServer and commodity hardware.

Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.

Perhaps we aren't agreeing at some points due to different markets. For example, I've used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked... Wasn't for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn't the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.

Leaving all those "political" points aside, let's talk technically :)

I've managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0

I'm really excited because it worked. So let's me explain what I've did.

First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP's of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.

Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD's are the physical connect from separate XS hosts to the shared storage. So it's not related to other hosts, it's exclusively to the host machine that I'm using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.

That's when things got interesting. To create the PBD, I do need the iSCSI working in the host, and consequently the multipath connection. So I done the process in the terminal with the following commands:

#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1

#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login

#Enable Multipath
multipath
multipath -ll

After all I was able to create the PBD with those new settings. The tricky part was to pass the required "device-config" values via the xe pbd-create command. I wasn't aware of the method, but after 4 tries I understood the process and typed the following command:

#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I'd like to understand your configuration better- have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?

Thanks,
Dave


Basically everything necessary is in there. The IP's, the name, the multipath link, the LUN values, everything. After this, I just plugged the PDB via the CLI:

#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342

And BOOM! It worked. As you can see in the screenshot.

To prove that's everything is working fine, I've created a new VM, installed FreeBSD 10 on it, done a pkg install xe-guest-utilities and requested a XenMotion operation. Oh boy, it worked:

Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms

Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0

I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It's just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.

What I ask now: a feedback, if possible.

Thanks in advance,
Vinícius.

PS: Sorry any english mistakes. It's almost 4AM here, and I was really anxious to report this in the list.


On Feb 14, 2015, at 8:14 PM, Tobias Kreidl <***@nau.edu><mailto:***@nau.edu> wrote:

Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.

The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.

That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.

Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.

Regards,
-=Tobias

On 2/14/2015 2:26 PM, Vinícius Ferrão wrote:

Hello guys,

I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.

At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it's only a interface to control some policies.

I've looked for help on ServerFault and FreeNAS Forums, since I'm running FreeNAS as iSCSI service. The discussion is going on those links:

http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/

It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn't apply.

If I understood correctly the SR is created with the appropriate PBD's. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD's from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.

Anyway, I'm opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that's a very common architecture and VMware is doing this on small pools! So let's do it equally and of course surpass them. :)

Thanks in advance,
Vinicius.
Vinícius Ferrão
2015-03-27 20:21:57 UTC
Permalink
Guys,

Sorry for the delayed answer, I was very busy on those days, but I was able to create the SR directly from the command-line without the needs of manually create the PBD's on each host. The resulting issued command was: (I splitted the command for better reading)

[***@xenserver1 ~]# xe sr-create \
host-uuid=c8927969-b7bf-4a0f-9725-035550381d9c \ <======== This is the host-uuid of the Pool Master.
device-config:multiSession="
192.168.10.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|
192.168.11.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|
192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|
192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|
192.168.30.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|
192.168.31.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" \
device-config:target=
192.168.10.1,
192.168.11.1,
192.168.20.1,
192.168.21.1,
192.168.30.1,
192.168.31.1 \
device-config:multihomelist=
192.168.10.1:3260,
192.168.11.1:3260,
192.168.20.1:3260,
192.168.21.1:3260,
192.168.30.1:3260,
192.168.31.1:3260
device-config:targetIQN=* \
device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 \
device-config:port=3260 \
name-label="FreeNAS iSCSI"
type=lvmoiscsi
shared=true

After some time the command answered, the UUID of the created SR: fa091aef-6645-6a77-6016-7b266a5e57e3

And in XenCenter everything was fine, the multipathed connections are made without any problem. See the attached screenshot for more details:
Loading Image... <Loading Image...

Do you guys think this last test is sufficient to implement this on XenCenter?

Thanks in advance,
Vinicius.
Post by Tobias J Kreidl
IMO. this would take modifying the XenCenter code to trigger running the SrRepairAction routine if it sees multiple, comma-separated entries in the target input field to make a new iSCSi conenction. That would be an ugly way to fix things, but might at least work.
If you try this and get one host conencted via XenCEnter and then do an SR Repair operation via XenCenter does that still work to get all the hosts conencted?
Sent: Friday, February 20, 2015 8:49 PM
Subject: Re: [xs-devel] Switchless shared iSCSI Network with /30 links
Any ideia on how to test this? I tried to create the pool for the first time using the string with all the six IP addresses, but XenCenter just fails without and error message, just an single X appears. The only thing I can provide at this moment is the /var/log/xensource.log; /var/log/audit.log; /var/log/messages and /var/log/SMlog file during the IQN/LUN phase, it's on the end of this message. The /var/log/SMlog appears to be the most valuable one.
I'm really dedicated on helping solving this issue, since I'm delaying the implementation of this farm just to keep the test scenario live!
Thanks,
PS: The logs...
*****/var/log/xensource.log*****
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|Async.SR.create R:b933cd75031b|dispatcher] spawning a new thread to handle the current task (trackid=8c8347a3f363fd40a9bd41eb1705050f)
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|audit] SR.create: name label = '__gui__'
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] SR.create name_label=__gui__ sm_config=[ ]
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|Async.SR.create R:b933cd75031b|storage_access] SR d403d0fc-4a14-7f99-fe92-dfc6301ed764 will be implemented by /services/SM/lvmoiscsi in VM OpaqueRef:47f2c383-de3e-730e-9836-d652d5eae646
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|mux] register SR d403d0fc-4a14-7f99-fe92-dfc6301ed764 (currently-registered = [ f31651b7-4219-dfe5-d871-4748c05e900c, ee47042b-76bf-5f62-2879-0457db8e892d, 12945887-3593-6c24-0b77-2efd62949415, d403d0fc-4a14-7f99-fe92-dfc6301ed764, 551b026c-6b94-5e08-ae67-3c76c15b8b44 ])
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|dummytaskhelper] task SR.create D:ebc6f0ef8713 created by task R:b933cd75031b
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|sm] SM lvmoiscsi sr_create sr=OpaqueRef:383eb943-faa8-1946-f169-6626b1996fe7 size=0
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Session.create trackid=70acc8c1d9618c45c2f07860af213730 pool=false uname= originator= is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|mscgen] xapi=>xapi [label="(XML)"];
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14339 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:7e7d99be1ccd created by task D:208b3f98f095
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|mscgen] smapiv2=>smapiv1 [label="sr_create"];
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:e234520429b7|api_effect] task.add_to_other_config
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|xenops] Event on VM 6dd2a08f-43a3-4f0c-b6c4-7e040829dc6a; resident_here = true
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|mscgen] xapi=>xenops [label="VM.exists"];
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|xenops] kernel is not in the whitelist: ignoring
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|mscgen] xapi=>xapi [label="(XML)"];
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14343 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:event.from D:ff5a707a4074 created by task D:4b65ef16de2c
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:7d79581bbf17|api_effect] task.add_to_other_config
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14344 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:27e54f4f8e11 created by task D:ebc6f0ef8713
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14345 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:fac3f1373b6e created by task D:ebc6f0ef8713
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14346 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:1713380916eb created by task D:ebc6f0ef8713
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:538ce8692691|api_effect] task.add_to_other_config
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Raised at sm_exec.ml:212.7-69 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Session.destroy trackid=70acc8c1d9618c45c2f07860af213730
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|backtrace] Raised at pervasiveext.ml:26.22-25 -> server_helpers.ml:76.11-23
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|dispatcher] Server_helpers.exec exception_handler: Got exception SR_BACKEND_FAILURE_96: [ ; The request is missing or has an incorrect target IQN parameter; <?xml version="1.0" ?> <iscsi-target-iqns> <TGT> <Index>
0 </Index>
<IPAddress> 192.168.10.1
</IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT>
<Index> 1
</Index> <IPAddress>
192.168.11.1 </IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT>
<Index> 2
</Index> <IPAddress>
192.168.20.1 </IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT>
<Index> 3
</Index> <IPAddress>
192.168.21.1 </IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT>
<Index> 4
</Index> <IPAddress>
192.168.30.1 </IPAddress>
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|dispatcher] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|backtrace] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|backtrace] Raised at storage_access.ml:161.15-44 -> server_helpers.ml:76.11-23
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|dispatcher] Server_helpers.exec exception_handler: Got exception INTERNAL_ERROR: [ Storage_interface.Backend_error(_) ]
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|dispatcher] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|backtrace] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] Raised at pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [error|xenserver1|14336|Async.SR.create R:b933cd75031b|storage_access] Re-raising as SR_BACKEND_FAILURE_96 [ ; The request is missing or has an incorrect target IQN parameter; <?xml version="1.0" ?> <iscsi-target-iqns>
<TGT> <Index>
0 </Index>
<IPAddress> 192.168.10.1
</IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT>
<Index> 1
</Index> <IPAddress>
192.168.11.1 </IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT>
<Index> 2
</Index> <IPAddress>
192.168.20.1 </IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT>
<Index> 3
</Index> <IPAddress>
192.168.21.1 </IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT>
<Index> 4
</Index> <IPAddress>
192.168.30.1 </IPAddress> <TargetIQN>iqn.2015-01.b
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|backtrace] Raised at xapi_sr.ml:225.9-10 -> server.ml:21898.82-267 -> rbac.ml:229.16-23
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|backtrace] Raised at rbac.ml:238.10-15 -> server_helpers.ml:79.11-41
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|dispatcher] Server_helpers.exec exception_handler: Got exception SR_BACKEND_FAILURE_96: [ ; The request is missing or has an incorrect target IQN parameter; <?xml version="1.0" ?> <iscsi-target-iqns> <TGT> <Index>
0 </Index>
<IPAddress> 192.168.10.1
</IPAddress> <TargetIQN>iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT>
<Index> 1
</Index> <IPAddress>
192.168.11.1 </IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT>
<Index> 2
</Index> <IPAddress>192.168.20.1
</IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT>
<Index> 3
</Index> <IPAddress>
192.168.21.1 </IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT>
<Index> 4
</Index> <IPAddress>
192.168.30.1 </IPAdd
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|dispatcher] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|backtrace] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336||xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336||xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|Async.SR.create R:202cf9100203|dispatcher] spawning a new thread to handle the current task (trackid=8c8347a3f363fd40a9bd41eb1705050f)
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|audit] SR.create: name label = '__gui__'
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] SR.create name_label=__gui__ sm_config=[ ]
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|Async.SR.create R:202cf9100203|storage_access] SR 207535ee-c14c-7b48-3869-96318c1c7877 will be implemented by /services/SM/lvmoiscsi in VM OpaqueRef:47f2c383-de3e-730e-9836-d652d5eae646
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|mux] register SR 207535ee-c14c-7b48-3869-96318c1c7877 (currently-registered = [ f31651b7-4219-dfe5-d871-4748c05e900c, ee47042b-76bf-5f62-2879-0457db8e892d, 207535ee-c14c-7b48-3869-96318c1c7877, 12945887-3593-6c24-0b77-2efd62949415, d403d0fc-4a14-7f99-fe92-dfc6301ed764, 551b026c-6b94-5e08-ae67-3c76c15b8b44 ])
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|dummytaskhelper] task SR.create D:7e6802e5fed5 created by task R:202cf9100203
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|sm] SM lvmoiscsi sr_create sr=OpaqueRef:5591eb47-1a35-9321-6aa4-bd8beb748cbf size=0
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Session.create trackid=836c7784c6fd5a42124fb4bd64aa2cd5 pool=false uname= originator= is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|mscgen] xapi=>xapi [label="(XML)"];
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14357 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:c14a1f2ccf30 created by task D:4a4af7d945d9
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|mscgen] smapiv2=>smapiv1 [label="sr_create"];
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:6f63db465b85|api_effect] task.add_to_other_config
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|xenops] Event on VM 6dd2a08f-43a3-4f0c-b6c4-7e040829dc6a; resident_here = true
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|mscgen] xapi=>xenops [label="VM.exists"];
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|xenops] kernel is not in the whitelist: ignoring
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|mscgen] xapi=>xapi [label="(XML)"];
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14359 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:event.from D:dd4a82303df3 created by task D:4b65ef16de2c
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:454990b8d01f|api_effect] task.add_to_other_config
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14360 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:513472992344 created by task D:7e6802e5fed5
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14361 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:4cb6c60153d2 created by task D:7e6802e5fed5
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14362 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:901d21392528 created by task D:7e6802e5fed5
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:f7f97c5c1dec|api_effect] task.add_to_other_config
Feb 21 01:41:17 xenserver1 xapi: [debug|xenserver1|14365 INET 0.0.0.0:80|Get RRD updates. D:8f822f0bc4c4|xapi] hand_over_connection GET /rrd_updates to /var/xapi/xcp-rrdd.forwarded
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Raised at forkhelpers.ml:187.30-76 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Raised at sm_exec.ml:190.10-100 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Raised at pervasiveext.ml:26.22-25 -> sm_exec.ml:173.23-1023 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [ info|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Session.destroy trackid=836c7784c6fd5a42124fb4bd64aa2cd5
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|backtrace] Raised at pervasiveext.ml:26.22-25 -> server_helpers.ml:76.11-23
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|dispatcher] Server_helpers.exec exception_handler: Got exception SR_BACKEND_FAILURE: [ non-zero exit; ; Traceback (most recent call last): File "/opt/xensource/sm/LVMoISCSISR", line 582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File "/opt/xensource/sm/SRCommand.py", line 343, in run sr = driver(cmd, cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line 142, in __init__ self.load(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line 156, in load self.iscsi = self.iscsiSRs[0] IndexError: list index out of range ]
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|dispatcher] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|backtrace] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|backtrace] Raised at storage_access.ml:161.15-44 -> server_helpers.ml:76.11-23
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|dispatcher] Server_helpers.exec exception_handler: Got exception INTERNAL_ERROR: [ Storage_interface.Backend_error(_) ]
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|dispatcher] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|backtrace] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] Raised at pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [error|xenserver1|14352|Async.SR.create R:202cf9100203|storage_access] Re-raising as SR_BACKEND_FAILURE [ non-zero exit; ; Traceback (most recent call last): File "/opt/xensource/sm/LVMoISCSISR", line 582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File "/opt/xensource/sm/SRCommand.py", line 343, in run sr = driver(cmd, cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line 142, in __init__ self.load(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line 156, in load self.iscsi = self.iscsiSRs[0] IndexError: list index out of range ]
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|backtrace] Raised at xapi_sr.ml:225.9-10 -> server.ml:21898.82-267 -> rbac.ml:229.16-23
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|backtrace] Raised at rbac.ml:238.10-15 -> server_helpers.ml:79.11-41
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|dispatcher] Server_helpers.exec exception_handler: Got exception SR_BACKEND_FAILURE: [ non-zero exit; ; Traceback (most recent call last): File "/opt/xensource/sm/LVMoISCSISR", line 582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File "/opt/xensource/sm/SRCommand.py", line 343, in run sr = driver(cmd, cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line 142, in __init__ self.load(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line 156, in load self.iscsi = self.iscsiSRs[0] IndexError: list index out of range ]
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|dispatcher] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|backtrace] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352||xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352||xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0 monitor|main|rrdd_stats] system stats: MemTotal: 3959600 KiB; MemFree: 3262140 KiB; Buffered: 33832 KiB; Cached: 291048 KiB; SwapTotal: 524284 KiB; SwapFree: 524284 KiB
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0 monitor|main|rrdd_stats] Clock drift: -0
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0 monitor|main|rrdd_stats] xcp-rrdd stats (n = 1): size: 155052 KiB; rss: 8368 KiB; data: 71620 KiB; stack: 136 KiB
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0 monitor|main|rrdd_stats] xapi stats (n = 2): size: 604328 KiB; rss: 101844 KiB; data: 451340 KiB; stack: 272 KiB
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0 monitor|main|rrdd_stats] xenopsd stats (n = 1): size: 278248 KiB; rss: 8308 KiB; data: 193692 KiB; stack: 136 KiB
***** /var/log/audit.log *****
Feb 21 01:41:01 xenserver1 xapi: [20150221T03:41:01.707Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:ee0292d7107f|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' '577acdd7-2b8c-b177-bf45-f8422c4ed7ca' 'OpaqueRef:b933cd75-031b-6ff0-5a0a-2341507af648')))
Feb 21 01:41:01 xenserver1 xapi: [20150221T03:41:01.747Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:ef6d521270fb|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' '577acdd7-2b8c-b177-bf45-f8422c4ed7ca' 'OpaqueRef:b933cd75-031b-6ff0-5a0a-2341507af648') ('value' '' 'e1aaff51-0da2-2745-cf25-ab034aea8a25' 'OpaqueRef:85f25443-719e-105d-35aa-a1c1043fdd8e')))
Feb 21 01:41:01 xenserver1 xapi: [20150221T03:41:01.787Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:43071305947a|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' '577acdd7-2b8c-b177-bf45-f8422c4ed7ca' 'OpaqueRef:b933cd75-031b-6ff0-5a0a-2341507af648')))
Feb 21 01:41:01 xenserver1 xapi: [20150221T03:41:01.885Z|audit|xenserver1|14336|Async.SR.create R:b933cd75031b|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:SR_BACKEND_FAILURE_96: [ ; The request is missing or has an incorrect target IQN parameter; <?xml version=\"1.0\" ?> <iscsi-target-iqns>
<TGT> <Index>
0 </Index>
<IPAddress> 192.168.10.1
</IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT>
<Index> 1
</Index> <IPAddress>
192.168.11.1 </IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT>
<Index> 2
</Index> <IPAddress>
192.168.20.1 </IPAddress> <TargetIQN>iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT>
<Index> 3
</Index> <IPAddress>
192.168.21.1 </IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT>
<Index>
Feb 21 01:41:02 xenserver1 xapi: [20150221T03:41:02.883Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.destroy D:5804968b2ed6|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.destroy' (('self' 'Async.SR.create' '577acdd7-2b8c-b177-bf45-f8422c4ed7ca' 'OpaqueRef:b933cd75-031b-6ff0-5a0a-2341507af648')))
Feb 21 01:41:06 xenserver1 xapi: [20150221T03:41:06.598Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:a09ea039fcd0|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' 'a0d8e973-9995-fb15-532e-f36ac5184c28' 'OpaqueRef:202cf910-0203-a943-9054-8d0e25a2b4a7')))
Feb 21 01:41:06 xenserver1 xapi: [20150221T03:41:06.638Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:b4dd3f0d946a|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' 'a0d8e973-9995-fb15-532e-f36ac5184c28' 'OpaqueRef:202cf910-0203-a943-9054-8d0e25a2b4a7') ('value' '' 'e1aaff51-0da2-2745-cf25-ab034aea8a25' 'OpaqueRef:85f25443-719e-105d-35aa-a1c1043fdd8e')))
Feb 21 01:41:06 xenserver1 xapi: [20150221T03:41:06.679Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:9b0a16a91401|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' 'a0d8e973-9995-fb15-532e-f36ac5184c28' 'OpaqueRef:202cf910-0203-a943-9054-8d0e25a2b4a7')))
Feb 21 01:41:17 xenserver1 xapi: [20150221T03:41:18.001Z|audit|xenserver1|14365 INET 0.0.0.0:80|handler:http/rrd_updates D:3d1bf1a8b00c|audit] ('trackid=bf92de30818eb264b5673ffe734ad369' 'LOCAL_SUPERUSER' '' 'ALLOWED' 'OK' 'HTTP' 'http/rrd_updates' ())
Feb 21 01:41:35 xenserver1 xapi: [20150221T03:41:35.155Z|audit|xenserver1|14352|Async.SR.create R:202cf9100203|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:SR_BACKEND_FAILURE: [ non-zero exit; ; Traceback (most recent call last): File \"/opt/xensource/sm/LVMoISCSISR\", line 582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File \"/opt/xensource/sm/SRCommand.py\", line 343, in run sr = driver(cmd, cmd.sr_uuid) File \"/opt/xensource/sm/SR.py\", line 142, in __init__ self.load(sr_uuid) File \"/opt/xensource/sm/LVMoISCSISR\", line 156, in load self.iscsi = self.iscsiSRs[0] IndexError: list index out of range ]' 'API' 'SR.create' (('host' 'xenserver1.local.iq.ufrj.br <http://xenserver1.local.iq.ufrj.br/>' 'c8927969-b7bf-4a0f-9725-035550381d9c' 'OpaqueRef:03239cf1-76cf-9c86-b927-56ea27cb6b94') ('name_label' '__gui__' '' '') ('name_description' 'SHOULD NEVER BE CREATED' '' '')))
Feb 21 01:41:35 xenserver1 xapi: [20150221T03:41:35.707Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.destroy D:7ff3ff7db08a|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.destroy' (('self' 'Async.SR.create' 'a0d8e973-9995-fb15-532e-f36ac5184c28' 'OpaqueRef:202cf910-0203-a943-9054-8d0e25a2b4a7')))
***** /var/log/messages *****
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|Async.SR.create R:b933cd75031b|dispatcher] spawning a new thread to handle the current task (trackid=8c8347a3f363fd40a9bd41eb1705050f)
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|Async.SR.create R:b933cd75031b|storage_access] SR d403d0fc-4a14-7f99-fe92-dfc6301ed764 will be implemented by /services/SM/lvmoiscsi in VM OpaqueRef:47f2c383-de3e-730e-9836-d652d5eae646
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Session.create trackid=70acc8c1d9618c45c2f07860af213730 pool=false uname= originator= is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Session.destroy trackid=70acc8c1d9618c45c2f07860af213730
Feb 21 01:41:01 xenserver1 xapi: [error|xenserver1|14336|Async.SR.create R:b933cd75031b|storage_access] Re-raising as SR_BACKEND_FAILURE_96 [ ; The request is missing or has an incorrect target IQN parameter; <?xml version="1.0" ?> <iscsi-target-iqns>
<TGT> <Index>
0 </Index>
<IPAddress> 192.168.10.1
</IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT>
<Index> 1
</Index> <IPAddress>
192.168.11.1 </IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT>
<Index> 2
</Index> <IPAddress>
192.168.20.1 </IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT>
<Index> 3
</Index> <IPAddress>
192.168.21.1 </IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT>
<Index> 4
</Index> <IPAddress>
192.168.30.1 </IPAddress> <TargetIQN>iqn.2015-01.b
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|Async.SR.create R:202cf9100203|dispatcher] spawning a new thread to handle the current task (trackid=8c8347a3f363fd40a9bd41eb1705050f)
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|Async.SR.create R:202cf9100203|storage_access] SR 207535ee-c14c-7b48-3869-96318c1c7877 will be implemented by /services/SM/lvmoiscsi in VM OpaqueRef:47f2c383-de3e-730e-9836-d652d5eae646
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Session.create trackid=836c7784c6fd5a42124fb4bd64aa2cd5 pool=false uname= originator= is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05155|bond|INFO|bond bond0: shift 21kB of load (with hash 35) from eth1 to eth0 (now carrying 233kB and 94kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05156|bond|INFO|bond bond0: shift 224kB of load (with hash 39) from eth1 to eth0 (now carrying 9kB and 318kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05157|bond|INFO|bond bond0: shift 0kB of load (with hash 9) from eth0 to eth1 (now carrying 318kB and 9kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05158|bond|INFO|bond bond0: shift 1kB of load (with hash 12) from eth0 to eth1 (now carrying 317kB and 10kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05159|bond|INFO|bond bond0: shift 1kB of load (with hash 14) from eth0 to eth1 (now carrying 316kB and 11kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05160|bond|INFO|bond bond0: shift 0kB of load (with hash 20) from eth0 to eth1 (now carrying 315kB and 11kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05161|bond|INFO|bond bond0: shift 0kB of load (with hash 21) from eth0 to eth1 (now carrying 315kB and 12kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05162|bond|INFO|bond bond0: shift 0kB of load (with hash 25) from eth0 to eth1 (now carrying 315kB and 12kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05163|bond|INFO|bond bond0: shift 15kB of load (with hash 46) from eth0 to eth1 (now carrying 300kB and 27kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05164|bond|INFO|bond bond0: shift 2kB of load (with hash 70) from eth0 to eth1 (now carrying 297kB and 30kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05165|bond|INFO|bond bond0: shift 1kB of load (with hash 84) from eth0 to eth1 (now carrying 295kB and 32kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05166|bond|INFO|bond bond0: shift 0kB of load (with hash 94) from eth0 to eth1 (now carrying 295kB and 32kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05167|bond|INFO|bond bond0: shift 2kB of load (with hash 112) from eth0 to eth1 (now carrying 293kB and 34kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05168|bond|INFO|bond bond0: shift 28kB of load (with hash 118) from eth0 to eth1 (now carrying 265kB and 62kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05169|bond|INFO|bond bond0: shift 5kB of load (with hash 160) from eth0 to eth1 (now carrying 259kB and 68kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05170|bond|INFO|bond bond0: shift 5kB of load (with hash 171) from eth0 to eth1 (now carrying 254kB and 73kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05171|bond|INFO|bond bond0: shift 3kB of load (with hash 206) from eth0 to eth1 (now carrying 251kB and 76kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05172|bond|INFO|bond bond0: shift 2kB of load (with hash 254) from eth0 to eth1 (now carrying 249kB and 78kB load, respectively)
Feb 21 01:41:28 xenserver1 ovs-vswitchd: ovs|05173|ofproto|INFO|xapi0: 29 flow_mods in the last 59 s (29 adds)
Feb 21 01:41:35 xenserver1 fe: 2498 (/opt/xensource/sm/LVMoISCSISR <methodCall><methodName>sr_create</methodName><...) exited with code 1
Feb 21 01:41:35 xenserver1 xapi: [ info|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Session.destroy trackid=836c7784c6fd5a42124fb4bd64aa2cd5
Feb 21 01:41:35 xenserver1 xapi: [error|xenserver1|14352|Async.SR.create R:202cf9100203|storage_access] Re-raising as SR_BACKEND_FAILURE [ non-zero exit; ; Traceback (most recent call last): File "/opt/xensource/sm/LVMoISCSISR", line 582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File "/opt/xensource/sm/SRCommand.py", line 343, in run sr = driver(cmd, cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line 142, in __init__ self.load(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line 156, in load self.iscsi = self.iscsiSRs[0] IndexError: list index out of range ]
Feb 21 01:41:56 xenserver1 xenstored: D3.14 rm attr/eth0
Feb 21 01:41:56 xenserver1 xenstored: D3 write attr/eth0/ip 10.7.0.243
Feb 21 01:41:56 xenserver1 xenstored: D3 write attr/eth0/ipv6/0/addr fe80::484b:acff:fe91:5b19
***** /var/log/SMlog *****
Feb 21 01:41:01 xenserver1 SM: [2444] _testHost: Testing host/port: 192.168.10.1,3260
Feb 21 01:41:01 xenserver1 SM: [2444] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] Raising exception [96, The request is missing or has an incorrect target IQN parameter]
Feb 21 01:41:06 xenserver1 SM: [2498] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] Raising exception [202, General backend error [opterr=rc: 21, stdout: , stderr: iscsiadm: No active sessions.
Feb 21 01:41:06 xenserver1 SM: [2498] ]]
Feb 21 01:41:06 xenserver1 SM: [2498] ['iscsiadm', '-m', 'session'] failed with (u'General backend error [opterr=rc: 21, stdout: , stderr: iscsiadm: No active sessions.\n]',)
Feb 21 01:41:06 xenserver1 SM: [2498] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] ['ls', '/sys/class/scsi_host', '-1', '--color=never']
Feb 21 01:41:06 xenserver1 SM: [2498] pread SUCCESS
Feb 21 01:41:06 xenserver1 SM: [2498] []
Feb 21 01:41:06 xenserver1 SM: [2498] PATHDICT: key 192.168.10.1:3260: {'path': '/dev/iscsi/*/192.168.10.1:3260', 'ipaddr': '192.168.10.1', 'port': 3260L}
Feb 21 01:41:06 xenserver1 SM: [2498] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] Discovery for IP 192.168.10.1 returned [('192.168.10.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.11.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.20.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.21.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.30.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.31.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1')]
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.10.1,3260
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.11.1,3260
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.20.1,3260
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.20.1) - (113, 'No route to host')
Feb 21 01:41:06 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:06 xenserver1 SM: [2498] Target Not reachable: (192.168.20.1:3260)
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.21.1,3260
Feb 21 01:41:07 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.21.1) - (113, 'No route to host')
Feb 21 01:41:07 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:07 xenserver1 SM: [2498] Target Not reachable: (192.168.21.1:3260)
Feb 21 01:41:07 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.30.1,3260
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.30.1) - timed out
Feb 21 01:41:09 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:09 xenserver1 SM: [2498] Target Not reachable: (192.168.30.1:3260)
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.31.1,3260
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.31.1) - (113, 'No route to host')
Feb 21 01:41:09 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:09 xenserver1 SM: [2498] Target Not reachable: (192.168.31.1:3260)
Feb 21 01:41:09 xenserver1 SM: [2498] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:09 xenserver1 SM: [2498] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:09 xenserver1 SM: [2498] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:09 xenserver1 SM: [2498] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:09 xenserver1 SM: [2498] Discovery for IP 192.168.11.1 returned [('192.168.10.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.11.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.20.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.21.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.30.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.31.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1')]
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.10.1,3260
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.11.1,3260
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.20.1,3260
Feb 21 01:41:11 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.20.1) - timed out
Feb 21 01:41:11 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:11 xenserver1 SM: [2498] Target Not reachable: (192.168.20.1:3260)
Feb 21 01:41:11 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.21.1,3260
Feb 21 01:41:11 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.21.1) - (113, 'No route to host')
Feb 21 01:41:11 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:11 xenserver1 SM: [2498] Target Not reachable: (192.168.21.1:3260)
Feb 21 01:41:11 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.30.1,3260
Feb 21 01:41:12 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.30.1) - (113, 'No route to host')
Feb 21 01:41:12 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:12 xenserver1 SM: [2498] Target Not reachable: (192.168.30.1:3260)
Feb 21 01:41:12 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.31.1,3260
Feb 21 01:41:14 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.31.1) - timed out
Feb 21 01:41:14 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:14 xenserver1 SM: [2498] Target Not reachable: (192.168.31.1:3260)
Feb 21 01:41:15 xenserver1 SM: [2498] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:15 xenserver1 SM: [2498] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:35 xenserver1 SM: [2498] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:35 xenserver1 SM: [2498] Raising exception [202, General backend error [opterr=rc: 21, stdout: , stderr: iscsiadm: cannot make connection to 192.168.20.1: No route to host
Feb 21 01:41:35 xenserver1 SM: [2498] iscsiadm: cannot make connection to 192.168.20.1: No route to host
Feb 21 01:41:35 xenserver1 last message repeated 3 times
Feb 21 01:41:35 xenserver1 SM: [2498] iscsiadm: connect to 192.168.20.1 timed out
Feb 21 01:41:35 xenserver1 SM: [2498] iscsiadm: connection login retries (reopen_max) 5 exceeded
Feb 21 01:41:35 xenserver1 SM: [2498] iscsiadm: No portals found
Feb 21 01:41:35 xenserver1 SM: [2498] ]]
Feb 21 01:41:35 xenserver1 SM: [2498] Raising exception [68, ISCSI login failed, verify CHAP credentials]
Feb 21 01:41:35 xenserver1 SM: [2498] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:35 xenserver1 SM: [2498] ***** LVHDoISCSISR.load: EXCEPTION SR.SROSError, ISCSI login failed, verify CHAP credentials
Feb 21 01:41:35 xenserver1 SM: [2498] File "/opt/xensource/sm/LVMoISCSISR", line 119, in load
Feb 21 01:41:35 xenserver1 SM: [2498] map = iscsilib.discovery(tgt_ip,iscsi.port,iscsi.chapuser,iscsi.chappassword,targetIQN=IQN)
Feb 21 01:41:35 xenserver1 SM: [2498] File "/opt/xensource/sm/iscsilib.py", line 156, in discovery
Feb 21 01:41:35 xenserver1 SM: [2498] raise xs_errors.XenError('ISCSILogin')
Feb 21 01:41:35 xenserver1 SM: [2498] File "/opt/xensource/sm/xs_errors.py", line 52, in __init__
Feb 21 01:41:35 xenserver1 SM: [2498] raise SR.SROSError(errorcode, errormessage)
Feb 21 01:41:35 xenserver1 SM: [2498]
Feb 21 01:41:54 xenserver1 updatempppathd: [7617] The garbage collection routine returned: 0
When I mentioned running the SrRepairAction procedure, I meant in conjunction with what is not happening in XenCenter as a follow-up to fix things.
So then, it would seem the piece remaining to resolve is what is missing in XenCenter that is not triggered. My guess is that it can perhaps only perform a connection using one IP address (or wildcarded address) and hence apparently doesn't understand what to do with anything involving a multiSession/multihome list?
It would be a useful diagnostic to see what is parsed out and processed if you pass in the whole multihome list of
192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260
to XenCenter and see what it ends up trying to do with it.
-=Tobias
Post by Vinícius Ferrão
Guys, another important thing that I've observed.
All the iscsiadm and multipath commands are useless. I've attached the same storage just issuing a specific PBD command from the Pool Master. So this simplifies the problem a lot.
xe pbd-create \
sr-uuid=f8d481b8-be10-922e-b875-91c4d73d341d \
host-uuid=44955863-f4a3-4770-bd7a-f4342dd7925a \
device-config:multiSession="192.168.30.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.31.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" \
device-config:target=192.168.30.1,192.168.31.1 \
device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 \
device-config:targetIQN=* \
device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 \
device-config:port=3260
The device-config attributes are sufficient to create all the connection.
Thanks,
Vinicius.
It may be possible to at least temporarily get this work by forcing the SrRepairAction procedure to be run.
I didn't have time yet, but wanted to compare what it does in sequence vs. what is done by default in the xapi_sr
program to see what is different. As far as XenCenter is concerned, perhaps if multiple IP target addresses are entered in the dialog box (as opposed to a single one or a wildcard entry), then a different means could be triggered to ensure that each host does a proper PBD creation and plug-in for that new SR.
-=Tobias
Post by Vinícius Ferrão
Dave,
I've added the "future" server to our pool. And happened exactly what you said. The configurations were cloned. But obviously it failed to clone the PDB's and the Network configuration for the iSCSI network.
1. The network configuration on added host should be "confirmed" to the user, and not done automatically, so the user can adapt is to specific needs.
2. After this the PDB creation should be done accordingly. If the network topology has changed by the user, so the PDBs must be created with this new topology.
In resume, I've three host at this moment on the pool. So I can do the HA tests once again, and I will report to the list the results. At this moment the only thing that failed is cloning the iSCSI network configurations and plugging the PDBs. Only the bounded NIC0+NIC1 was created on the added host.
Thanks in advance,
Vinícius.
Post by Dave Scott
Regarding Point 3) I believe what is meant is that you should test having more than one SR that you are connecting to using the same IP addresses for the target array. You know it works fine to create one SR, so can you add additional ones the same way and have them all live together in harmony? :-)
As to the sr-create command, the host-uuid is an optional parameter and hence does not need to be named for the pool master or any other. The only required parameters are the name-label and the SR type. Have you tried that to see if the behavior is any different compared to when you do supply a host name?
I believe the underlying Xapi data model can express what you want, but there are 1 or 2 places where Xapi will helpfully create PBDs for you by cloning the master, which you just need to be aware of.
For example you should be able to use something like
$ xe sr-create .. host-uuid=<any host> shared=false
Setting “shared=false” initially should avoid Xapi ‘helpfully’ pre-creating identical PBDs for every host with the same configuration[1]. You should end up with an SR attached to a single host.
$ xe sr-param-set uuid= shared=true
$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...
$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...
[ as an aside, it’s obviously not great that the PBDs you get depend on exactly when you declare the SR is shared. This has happened because the original design had all PBDs being distinct, and the notion of ‘copying the master’ was added relatively late ]
Whenever a new host is added to the pool, Xapi will clone the master’s PBD for you— you would have to destroy it and re-create it. I suppose cloning the master’s configuration is as good a guess as any in this situation :-)
Dave
[1] https://github.com/xapi-project/xen-api/blob/b6f28c52fd9726553968b410e6b8cced5401f385/ocaml/xapi/xapi_sr.ml#L209 <https://github.com/xapi-project/xen-api/blob/b6f28c52fd9726553968b410e6b8cced5401f385/ocaml/xapi/xapi_sr.ml#L209>
-=Tobias
Post by Vinícius Ferrão
Hello guys,
Answering about all the questions and issues. For those who aren't understanding the network topology, I've made a drawing of the solution. The image is here: https://www.dropbox.com/s/6zwerk6igcyqou3/UFRJ-IQ-VirtualPool.jpg?dl=0 <https://www.dropbox.com/s/6zwerk6igcyqou3/UFRJ-IQ-VirtualPool.jpg?dl=0>
192.168.10.1
192.168.11.1
192.168.20.1
192.168.21.1
The convention to this numbering scheme was: 192.168.{XenServerNumber}{InterfaceNumber}.{Storage(1)/Host(2)}.
192.168.10.2 (XenServer #1 - Interface #0)
192.168.11.2 (XenServer #1 - Interface #1)
192.168.20.2 (XenServer #2 - Interface #0)
192.168.21.2 (XenServer #2 - Interface #1)
Hope this completely clarifies the architecture/topology of the iSCSI network.
Restarting both hosts on the same time. [OK]
Restarting the second host. [OK]
Restarting the pool master. [OK]
2. HA is working too. It fences only the damaged host. I even tried a VM Failover. HA happened without any problem and the lost VM has been restarted on other server. To do the HA test, I've power cycled the host, simulated a hardware crash scenario. A cool picture: https://www.dropbox.com/s/g7h7rljk8vh97rg/Screenshot%202015-02-18%2015.45.41.png?dl=0 <https://www.dropbox.com/s/g7h7rljk8vh97rg/Screenshot%202015-02-18%2015.45.41.png?dl=0>
3. I haven't understood this test. You are asking to create a new LUN and add a new Shared Repository? It's not clear what you're saying with "independent SR".
4. Storage XenMotion works too. I've moved a running VM from local storage (on the Host #2) to the iSCSI pool without any problem. The reverse process works too.
5. I will test this tomorrow, when I will got physical access to the servers (carnival holiday here in BR). But about this specific test, I can't see any problem with plain Xen Motion, since the data is copied over the management network and not the iSCSI network. But I can see a good point when dealing with Storage XenMotion.
Finally about the changes on XenCenter or XenServer/XAPI itself. One thing to prove this point is to create the SR via the CLI. If I would be able to do this, the problem should definitely be on XenCenter. As I said before, I've created a broken Software iSCSI SR via XenCenter and them mapped the iSCSI interfaces, enabled multipath and created the PBD over the CLI.
The iSCSI and Multipath should be done in the physical host. But the PBD creation can be done without any problem on the Pool Master.
device-config: name-label= shared= type=
In this case sm-config and device-config are the problematic ones. Assuming that the host-uuid is always the pool master.
Thanks in advance,
Vinícius.
I’ll add in a few more tests….
1. Host restart. Does the normal PBD plug mechanism work with such a configuration
2. HA. While HA with two hosts isn’t a supported configuration, in this model it might work properly. If you remove the network cables for a single host, does it fence, or do both hosts fence?
3. Multiple SRs. If you have multiple LUNs on the FreeNAS exported and then used as independent SRs, does everything work as it should?
4. Storage XenMotion. With a vdi on one SR presented with the FreeNAS, copy it to any another storage option
5. Motion during path failure. With one path down on a given host, does XenMotion and Storage XenMotion still function correctly?
For those who are questioning the value of such a configuration, this is in essence a “cluster in a box” solution. These configurations were very popular 10-15 years ago as highly available compute infrastructure was quite expensive back then. Today we can accomplish the same things we did back then using virtualization, but “cluster in a box” can always be valuable for those with limited IT skills or for smaller organizations wanting least cost with fewest points of failure.
-tim
Sent: Monday, February 16, 2015 7:24 PM
To: Tobias Kreidl
Subject: Re: [xs-devel] Switchless shared iSCSI Network with /30 links
Tobias,
I've done more tests about the SR creation over XenCenter.
All the process definitely occurs only on the pool master. Even if the specified networks are not part of the networks of the host. The connection is tried on the default route.
192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1
The target IQN is discovered without any problem, since the iSCSI Storage reports the LUNs on any interface.
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 192.168.10.2:55870 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52654 192.168.11.1:3260 TIME_WAIT
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.11.2:52656 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
******************
tcp 0 1 10.7.0.101:56481 192.168.21.1:3260 SYN_SENT
******************
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
******************
tcp 0 1 10.7.0.101:52787 192.168.30.1:3260 SYN_SENT
******************
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
If you're wondering the 192.168.30.1 is reported because in the future we will add another XS host to the pool. The point here is that XenCenter should not lookup for this network since it isn't specified during the Software iSCSI creation. On my understanding it should only lookup for the specified addresses.
tcp 0 48 10.7.0.102:22 192.168.172.1:58046 ESTABLISHED
udp 0 0 192.168.21.2:123 0.0.0.0:*
udp 0 0 192.168.20.2:123 0.0.0.0:*
As you said, perhaps propagating the initial connection through the host will solve the problem without modifying the XenCenter interface. A more sophisticated approach would be query the network-list over XAPI, retrieve the PIFs associated to those networks and them just check the IP addresses on the SR creation request and match them with the equivalent PIFs.
uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa
name-label ( RW): iSCSI #0
PIF-uuids (SRO): 3baa8436-feca-cd04-3173-5002d9ffc66b; 362d63cb-647a-465f-6b81-1b59cd3b1f5d
MTU ( RW): 9000
bridge ( RO): xenbr2
other-config (MRW): automatic: true
default-locking-mode ( RW): unlocked
uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5c:64
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
host-name-label ( RO): xenserver2.local.iq.ufrj.br <http://xenserver2.local.iq.ufrj.br/>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.20.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5a:1c
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c
host-name-label ( RO): xenserver1.local.iq.ufrj.br <http://xenserver1.local.iq.ufrj.br/>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.10.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
So as you can see, we have the IP addresses and the network mask. Which is sufficient to an efficient and precise algorithm of SR creation.
I thinks this clarifies everything we discussed until now. A patch on XenCenter appears to be really viable.
Many thanks,
Vinicius.
Yes, correct. The MD36xx looks like two separate controller units, each acting as a separate device. For a standalone server or some storage devices, that's right that you need a separate subnet for each connection.
Because a single connection isn't seeing the broadcast from the other connectors at the time of the initial connection from one host, the pool is unaware of the others. The "repair" operation causes this to be conducted from each host, and as I guessed, then fixes the SR connectivity for the entire pool. I am speculating that the request for a new connection of this type via XenCenter could perhaps be solved by propagating that initial connection process to the rest of the hosts in the pool and then rescan the SR.
-=Tobias
Hi Tobias,
As for I know, the Dell MD36xx is a dual controller based storage. So it's basically "two computers". Considering this with the added knowledge that I've about TCP networking there's a golden rule that says: only one IP address on a given subnet on a single computer. So I can't use the same network on my case. That's why I use point-to-point links too.
Since I've only one machine acting as a storage server (It's a Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do need four separate networks. FreeNAS is FreeBSD based, and BSD enforces this rule by default. I can't have two IP addresses of the same subnet on same machine, even if I manually configure it.
If this wasn't bad TCP network practice, I would be able to configure the SR via XenCenter, since it will "find" the same network on the others hosts without problem.
About the XenCenter, it really appears to be only a missing function on the software. If I was a good programmer I would try to implement this. But this is beyond my knowledge... :(
Thanks once again,
Vinícius.
No reason to necessarily use four separate subnets. You could have 192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and 2.2 on the other controller (on, for example, an MD36xx that is standard). If however it is something like an MD32xx which recommends four separate subets, then of course, use four. One should follow the vendor's specifications.
I will respond in more detail later when I get a chance, Vinícius, but after a quick read, I agree with your findings from your other email. It would appear that the initial connection is indeed evidently a missing function in XenCenter and perhaps all that is needed to make this work "out of the box". As to the wildcard discovery, it's generally preferred as it allows the host to figure out the path on its own. The only reason I could see not using a wildcard discovery would be if there were any chance for overlaps and ambiguous connections.
Regards,
--Tobias
Hello Dave,
+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+
192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30
192.168.10.2/30
192.168.11.2/30
192.168.20.2/30
192.168.21.2/30
That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0 <https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0>
Cheers,
Vinícius.
Hi,
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked… Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0 <https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0>
I’m really excited because it worked. So let’s me explain what I’ve did.
First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I’d like to understand your configuration better— have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?
Thanks,
Dave
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0 <https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0>
I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch <http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch>
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/ <https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/>
It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Tobias Kreidl
2015-03-27 20:47:48 UTC
Permalink
Hi, Vinícius!
There would still need to be some behind-the-scenes work done. In
essence, I believe what would work would be to change XenCenter so that
you can input multiple iSCSI targets and have it parse them all out and
propagate the connections accordingly to be initiated on all hosts
within the pool.

Perhaps another place to post this would be also on the Xen development
blog at:
https://xenorg.uservoice.com/forums/172169-xen-development as typically
a lot of work gets done and implemented there before it is absorbed by
XenServer itself.

Certainly, one would need look at the XenCenter source code; at present,
it's pretty clear you cannot enter multiple iSCSI targets in the dialog
box, but that's just the first thing that would have to be handled.

It's great, though, that this does all work from the command line and
consistently, so that is an important first step.

Regards,
-=Tobias
Post by Vinícius Ferrão
Guys,
Sorry for the delayed answer, I was very busy on those days, but I was
able to create the SR directly from the command-line without the needs
of manually create the PBD's on each host. The resulting issued
command was: (I splitted the command for better reading)
host-uuid=c8927969-b7bf-4a0f-9725-035550381d9c \ <======== This is the
host-uuid of the Pool Master.
device-config:multiSession="
192.168.10.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|
192.168.11.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|
192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|
192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|
192.168.30.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|
192.168.31.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" \
device-config:target=
192.168.10.1,
192.168.11.1,
192.168.20.1,
192.168.21.1,
192.168.30.1,
192.168.31.1 \
device-config:multihomelist=
192.168.10.1:3260,
192.168.11.1:3260,
192.168.20.1:3260,
192.168.21.1:3260,
192.168.30.1:3260,
192.168.31.1:3260
device-config:targetIQN=* \
device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 \
device-config:port=3260 \
name-label="FreeNAS iSCSI"
type=lvmoiscsi
shared=true
fa091aef-6645-6a77-6016-7b266a5e57e3
And in XenCenter everything was fine, the multipathed connections are
https://www.dropbox.com/s/lely8w8gxndjz7g/Screenshot%202015-03-27%2017.16.25.png?dl=0
Do you guys think this last test is sufficient to implement this on XenCenter?
Thanks in advance,
Vinicius.
Post by Tobias J Kreidl
IMO. this would take modifying the XenCenter code to trigger running
the SrRepairAction routine if it sees multiple, comma-separated
entries in the target input field to make a new iSCSi conenction.
That would be an ugly way to fix things, but might at least work.
If you try this and get one host conencted via XenCEnter and then do
an SR Repair operation via XenCenter does that still work to get all
the hosts conencted?
------------------------------------------------------------------------
*
*Sent:*Friday, February 20, 2015 8:49 PM
*Subject:*Re: [xs-devel] Switchless shared iSCSI Network with /30 links
Any ideia on how to test this? I tried to create the pool for the
first time using the string with all the six IP addresses, but
XenCenter just fails without and error message, just an single X
appears. The only thing I can provide at this moment is the
/var/log/xensource.log; /var/log/audit.log; /var/log/messages and
/var/log/SMlog file during the IQN/LUN phase, it's on the end of this
message. The /var/log/SMlog appears to be the most valuable one.
I'm really dedicated on helping solving this issue, since I'm
delaying the implementation of this farm just to keep the test
scenario live!
Thanks,
PS: The logs...
*****/var/log/xensource.log*****
Feb 21 01:41:01 xenserver1 xapi: [
info|xenserver1|14336|Async.SR.create R:b933cd75031b|dispatcher]
spawning a new thread to handle the current task
(trackid=8c8347a3f363fd40a9bd41eb1705050f)
[debug|xenserver1|14336|Async.SR.create R:b933cd75031b|audit]
SR.create: name label = '__gui__'
[debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi]
SR.create name_label=__gui__ sm_config=[ ]
Feb 21 01:41:01 xenserver1 xapi: [
info|xenserver1|14336|Async.SR.create R:b933cd75031b|storage_access]
SR d403d0fc-4a14-7f99-fe92-dfc6301ed764 will be implemented by
/services/SM/lvmoiscsi in VM
OpaqueRef:47f2c383-de3e-730e-9836-d652d5eae646
[debug|xenserver1|14336|Async.SR.create R:b933cd75031b|mux] register
SR d403d0fc-4a14-7f99-fe92-dfc6301ed764 (currently-registered = [
f31651b7-4219-dfe5-d871-4748c05e900c,
ee47042b-76bf-5f62-2879-0457db8e892d,
12945887-3593-6c24-0b77-2efd62949415,
d403d0fc-4a14-7f99-fe92-dfc6301ed764, 551b026c-6b94-5e08-ae67-3c76c15b8b44
])
[debug|xenserver1|14336|Async.SR.create
R:b933cd75031b|dummytaskhelper] task SR.create D:ebc6f0ef8713 created
by task R:b933cd75031b
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create
D:ebc6f0ef8713|sm] SM lvmoiscsi
sr_create sr=OpaqueRef:383eb943-faa8-1946-f169-6626b1996fe7 size=0
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|sm_exec
D:208b3f98f095|xapi]
Session.create trackid=70acc8c1d9618c45c2f07860af213730 pool=false
uname= originator= is_local_superuser=true
auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec
D:208b3f98f095|mscgen] xapi=>xapi [label="(XML)"];
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14339 UNIX
/var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid
D:7e7d99be1ccd created by task D:208b3f98f095
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec
D:208b3f98f095|mscgen] smapiv2=>smapiv1 [label="sr_create"];
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|13421 INET
0.0.0.0:80|dispatch:task.add_to_other_config D:e234520429b7|api_effect]
task.add_to_other_config
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|429|xapi events
D:4b65ef16de2c|xenops] Event on
VM 6dd2a08f-43a3-4f0c-b6c4-7e040829dc6a; resident_here = true
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|429|xapi events
D:4b65ef16de2c|mscgen] xapi=>xenops [label="VM.exists"];
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|429|xapi events
D:4b65ef16de2c|xenops] kernel is not in the whitelist: ignoring
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|429|xapi events
D:4b65ef16de2c|mscgen] xapi=>xapi [label="(XML)"];
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14343 UNIX
/var/xapi/xapi||dummytaskhelper] task dispatch:event.from
D:ff5a707a4074 created by task D:4b65ef16de2c
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|13421 INET
0.0.0.0:80|dispatch:task.add_to_other_config D:7d79581bbf17|api_effect]
task.add_to_other_config
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14344 UNIX
/var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config
D:27e54f4f8e11 created by task D:ebc6f0ef8713
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14345 UNIX
/var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config
D:fac3f1373b6e created by task D:ebc6f0ef8713
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14346 UNIX
/var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config
D:1713380916eb created by task D:ebc6f0ef8713
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|13421 INET
0.0.0.0:80|dispatch:task.add_to_other_config D:538ce8692691|api_effect]
task.add_to_other_config
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec
D:208b3f98f095|xapi] Raised at sm_exec.ml:212.7-69 ->
pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|sm_exec
D:208b3f98f095|xapi]
Session.destroy trackid=70acc8c1d9618c45c2f07860af213730
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec
D:208b3f98f095|backtrace] Raised at pervasiveext.ml:26.22-25 ->
server_helpers.ml:76.11-23
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec
D:208b3f98f095|dispatcher] Server_helpers.exec exception_handler: Got
exception SR_BACKEND_FAILURE_96: [ ; The request is missing or has an
incorrect target IQN parameter; <?xml version="1.0" ?>
<iscsi-target-iqns> <TGT> <Index> 0 </Index> <IPAddress> 192.168.10.1
</IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT>
<TGT> <Index> 1 </Index> <IPAddress> 192.168.11.1 </IPAddress>
<TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT> <Index> 2 </Index> <IPAddress> 192.168.20.1
</IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT>
<TGT> <Index> 3 </Index> <IPAddress> 192.168.21.1 </IPAddress>
<TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT> <Index> 4 </Index> <IPAddress> 192.168.30.1
</IPAddress>
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec
D:208b3f98f095|dispatcher] Raised at hashtbl.ml:97.23-32 ->
debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec
D:208b3f98f095|backtrace] Raised at hashtbl.ml:97.23-32 ->
debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec
D:208b3f98f095|xapi] Raised at server_helpers.ml:94.14-15 ->
pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec
D:208b3f98f095|xapi] Raised at pervasiveext.ml:26.22-25 ->
pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create
D:ebc6f0ef8713|xapi] Raised at pervasiveext.ml:26.22-25 ->
pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create
D:ebc6f0ef8713|backtrace] Raised at storage_access.ml:161.15-44 ->
server_helpers.ml:76.11-23
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create
D:ebc6f0ef8713|dispatcher] Server_helpers.exec exception_handler: Got
exception INTERNAL_ERROR: [ Storage_interface.Backend_error(_) ]
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create
D:ebc6f0ef8713|dispatcher] Raised at hashtbl.ml:97.23-32 ->
debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create
D:ebc6f0ef8713|backtrace] Raised at hashtbl.ml:97.23-32 ->
debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create
D:ebc6f0ef8713|xapi] Raised at server_helpers.ml:94.14-15 ->
pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create
D:ebc6f0ef8713|xapi] Raised at pervasiveext.ml:26.22-25 ->
pervasiveext.ml:22.2-9
[debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] Raised
at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
[debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] Raised
at pervasiveext.ml:22.2-9
[error|xenserver1|14336|Async.SR.create
R:b933cd75031b|storage_access] Re-raising as SR_BACKEND_FAILURE_96 [
; The request is missing or has an incorrect target IQN parameter;
<?xml version="1.0" ?> <iscsi-target-iqns> <TGT> <Index> 0 </Index>
<IPAddress> 192.168.10.1 </IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT>
<TGT> <Index> 1 </Index> <IPAddress> 192.168.11.1 </IPAddress>
<TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT> <Index> 2 </Index> <IPAddress> 192.168.20.1
</IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT>
<TGT> <Index> 3 </Index> <IPAddress> 192.168.21.1 </IPAddress>
<TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT> <Index> 4 </Index> <IPAddress> 192.168.30.1
</IPAddress> <TargetIQN>iqn.2015-01.b
[debug|xenserver1|14336|Async.SR.create R:b933cd75031b|backtrace]
Raised at xapi_sr.ml:225.9-10 -> server.ml:21898.82-267 ->
rbac.ml:229.16-23
[debug|xenserver1|14336|Async.SR.create R:b933cd75031b|backtrace]
Raised at rbac.ml:238.10-15 -> server_helpers.ml:79.11-41
[debug|xenserver1|14336|Async.SR.create
R:b933cd75031b|dispatcher] Server_helpers.exec exception_handler: Got
exception SR_BACKEND_FAILURE_96: [ ; The request is missing or has an
incorrect target IQN parameter; <?xml version="1.0" ?>
<iscsi-target-iqns> <TGT> <Index> 0 </Index> <IPAddress> 192.168.10.1
</IPAddress>
<TargetIQN>iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT> <Index> 1 </Index> <IPAddress> 192.168.11.1
</IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT>
<TGT> <Index> 2 </Index> <IPAddress>192.168.20.1 </IPAddress>
<TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT> <Index> 3 </Index> <IPAddress> 192.168.21.1
</IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT>
<TGT> <Index> 4 </Index> <IPAddress> 192.168.30.1 </IPAdd
[debug|xenserver1|14336|Async.SR.create R:b933cd75031b|dispatcher]
Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
[debug|xenserver1|14336|Async.SR.create R:b933cd75031b|backtrace]
Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
[debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] Raised
at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
[debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] Raised
at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336||xapi]
Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336||xapi]
Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:06 xenserver1 xapi: [
info|xenserver1|14352|Async.SR.create R:202cf9100203|dispatcher]
spawning a new thread to handle the current task
(trackid=8c8347a3f363fd40a9bd41eb1705050f)
[debug|xenserver1|14352|Async.SR.create R:202cf9100203|audit]
SR.create: name label = '__gui__'
[debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi]
SR.create name_label=__gui__ sm_config=[ ]
Feb 21 01:41:06 xenserver1 xapi: [
info|xenserver1|14352|Async.SR.create R:202cf9100203|storage_access]
SR 207535ee-c14c-7b48-3869-96318c1c7877 will be implemented by
/services/SM/lvmoiscsi in VM
OpaqueRef:47f2c383-de3e-730e-9836-d652d5eae646
[debug|xenserver1|14352|Async.SR.create R:202cf9100203|mux] register
SR 207535ee-c14c-7b48-3869-96318c1c7877 (currently-registered = [
f31651b7-4219-dfe5-d871-4748c05e900c,
ee47042b-76bf-5f62-2879-0457db8e892d,
207535ee-c14c-7b48-3869-96318c1c7877,
12945887-3593-6c24-0b77-2efd62949415, d403d0fc-4a14-7f99-fe92-dfc6301ed764,
551b026c-6b94-5e08-ae67-3c76c15b8b44 ])
[debug|xenserver1|14352|Async.SR.create
R:202cf9100203|dummytaskhelper] task SR.create D:7e6802e5fed5 created
by task R:202cf9100203
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|SR.create
D:7e6802e5fed5|sm] SM lvmoiscsi
sr_create sr=OpaqueRef:5591eb47-1a35-9321-6aa4-bd8beb748cbf size=0
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|sm_exec
D:4a4af7d945d9|xapi]
Session.create trackid=836c7784c6fd5a42124fb4bd64aa2cd5 pool=false
uname= originator= is_local_superuser=true
auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|sm_exec
D:4a4af7d945d9|mscgen] xapi=>xapi [label="(XML)"];
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14357 UNIX
/var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid
D:c14a1f2ccf30 created by task D:4a4af7d945d9
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|sm_exec
D:4a4af7d945d9|mscgen] smapiv2=>smapiv1 [label="sr_create"];
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|13421 INET
0.0.0.0:80|dispatch:task.add_to_other_config D:6f63db465b85|api_effect]
task.add_to_other_config
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|429|xapi events
D:4b65ef16de2c|xenops] Event on
VM 6dd2a08f-43a3-4f0c-b6c4-7e040829dc6a; resident_here = true
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|429|xapi events
D:4b65ef16de2c|mscgen] xapi=>xenops [label="VM.exists"];
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|429|xapi events
D:4b65ef16de2c|xenops] kernel is not in the whitelist: ignoring
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|429|xapi events
D:4b65ef16de2c|mscgen] xapi=>xapi [label="(XML)"];
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14359 UNIX
/var/xapi/xapi||dummytaskhelper] task dispatch:event.from
D:dd4a82303df3 created by task D:4b65ef16de2c
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|13421 INET
0.0.0.0:80|dispatch:task.add_to_other_config D:454990b8d01f|api_effect]
task.add_to_other_config
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14360 UNIX
/var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config
D:513472992344 created by task D:7e6802e5fed5
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14361 UNIX
/var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config
D:4cb6c60153d2 created by task D:7e6802e5fed5
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14362 UNIX
/var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config
D:901d21392528 created by task D:7e6802e5fed5
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|13421 INET
0.0.0.0:80|dispatch:task.add_to_other_config D:f7f97c5c1dec|api_effect]
task.add_to_other_config
Feb 21 01:41:17 xenserver1 xapi: [debug|xenserver1|14365 INET
0.0.0.0:80|Get RRD updates. D:8f822f0bc4c4|xapi] hand_over_connection
GET /rrd_updates to /var/xapi/xcp-rrdd.forwarded
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec
D:4a4af7d945d9|xapi] Raised at forkhelpers.ml:187.30-76 ->
pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec
D:4a4af7d945d9|xapi] Raised at sm_exec.ml:190.10-100 ->
pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec
D:4a4af7d945d9|xapi] Raised at pervasiveext.ml:26.22-25 ->
sm_exec.ml:173.23-1023 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [ info|xenserver1|14352|sm_exec
D:4a4af7d945d9|xapi]
Session.destroy trackid=836c7784c6fd5a42124fb4bd64aa2cd5
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec
D:4a4af7d945d9|backtrace] Raised at pervasiveext.ml:26.22-25 ->
server_helpers.ml:76.11-23
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec
D:4a4af7d945d9|dispatcher] Server_helpers.exec exception_handler: Got
exception SR_BACKEND_FAILURE: [ non-zero exit; ; Traceback
(most recent call last): File "/opt/xensource/sm/LVMoISCSISR", line
582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File
"/opt/xensource/sm/SRCommand.py", line 343, in run sr =
driver(cmd, cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line 142,
in __init__
self.load(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line
156, in load self.iscsi = self.iscsiSRs[0] IndexError: list
index out of range ]
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec
D:4a4af7d945d9|dispatcher] Raised at hashtbl.ml:93.19-28 ->
debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec
D:4a4af7d945d9|backtrace] Raised at hashtbl.ml:93.19-28 ->
debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec
D:4a4af7d945d9|xapi] Raised at server_helpers.ml:94.14-15 ->
pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec
D:4a4af7d945d9|xapi] Raised at pervasiveext.ml:26.22-25 ->
pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create
D:7e6802e5fed5|xapi] Raised at pervasiveext.ml:26.22-25 ->
pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create
D:7e6802e5fed5|backtrace] Raised at storage_access.ml:161.15-44 ->
server_helpers.ml:76.11-23
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create
D:7e6802e5fed5|dispatcher] Server_helpers.exec exception_handler: Got
exception INTERNAL_ERROR: [ Storage_interface.Backend_error(_) ]
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create
D:7e6802e5fed5|dispatcher] Raised at hashtbl.ml:93.19-28 ->
debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create
D:7e6802e5fed5|backtrace] Raised at hashtbl.ml:93.19-28 ->
debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create
D:7e6802e5fed5|xapi] Raised at server_helpers.ml:94.14-15 ->
pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create
D:7e6802e5fed5|xapi] Raised at pervasiveext.ml:26.22-25 ->
pervasiveext.ml:22.2-9
[debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] Raised
at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
[debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] Raised
at pervasiveext.ml:22.2-9
[error|xenserver1|14352|Async.SR.create
R:202cf9100203|storage_access] Re-raising as SR_BACKEND_FAILURE [
non-zero exit; ; Traceback (most recent call
last): File "/opt/xensource/sm/LVMoISCSISR", line 582, in ?
SRCommand.run(LVHDoISCSISR,
DRIVER_INFO) File "/opt/xensource/sm/SRCommand.py", line 343, in
run sr = driver(cmd,
cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line 142, in __init__
self.load(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line
156, in load self.iscsi = self.iscsiSRs[0] IndexError: list
index out of range ]
[debug|xenserver1|14352|Async.SR.create R:202cf9100203|backtrace]
Raised at xapi_sr.ml:225.9-10 -> server.ml:21898.82-267 ->
rbac.ml:229.16-23
[debug|xenserver1|14352|Async.SR.create R:202cf9100203|backtrace]
Raised at rbac.ml:238.10-15 -> server_helpers.ml:79.11-41
[debug|xenserver1|14352|Async.SR.create
R:202cf9100203|dispatcher] Server_helpers.exec exception_handler: Got
exception SR_BACKEND_FAILURE: [ non-zero exit; ; Traceback
(most recent call last): File "/opt/xensource/sm/LVMoISCSISR", line
582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File
"/opt/xensource/sm/SRCommand.py", line 343, in run sr =
driver(cmd, cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line 142,
in __init__
self.load(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line
156, in load self.iscsi = self.iscsiSRs[0] IndexError: list
index out of range ]
[debug|xenserver1|14352|Async.SR.create R:202cf9100203|dispatcher]
Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
[debug|xenserver1|14352|Async.SR.create R:202cf9100203|backtrace]
Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
[debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] Raised
at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
[debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] Raised
at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352||xapi]
Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352||xapi]
Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0
monitor|main|rrdd_stats] system stats: MemTotal: 3959600 KiB;
MemFree: 3262140 KiB; Buffered: 33832 KiB; Cached: 291048 KiB;
SwapTotal: 524284 KiB; SwapFree: 524284 KiB
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0
monitor|main|rrdd_stats] Clock drift: -0
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0
monitor|main|rrdd_stats] xcp-rrdd stats (n = 1): size: 155052 KiB;
rss: 8368 KiB; data: 71620 KiB; stack: 136 KiB
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0
101844 KiB; data: 451340 KiB; stack: 272 KiB
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0
monitor|main|rrdd_stats] xenopsd stats (n = 1): size: 278248 KiB;
rss: 8308 KiB; data: 193692 KiB; stack: 136 KiB
***** /var/log/audit.log *****
[20150221T03:41:01.707Z|audit|xenserver1|13421 INET
0.0.0.0:80|task.add_to_other_config
D:ee0292d7107f|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f'
'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API'
'task.add_to_other_config' (('self' 'Async.SR.create'
'577acdd7-2b8c-b177-bf45-f8422c4ed7ca'
'OpaqueRef:b933cd75-031b-6ff0-5a0a-2341507af648')))
[20150221T03:41:01.747Z|audit|xenserver1|13421 INET
0.0.0.0:80|task.add_to_other_config
D:ef6d521270fb|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f'
'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API'
'task.add_to_other_config' (('self' 'Async.SR.create'
'577acdd7-2b8c-b177-bf45-f8422c4ed7ca'
'OpaqueRef:b933cd75-031b-6ff0-5a0a-2341507af648') ('value' ''
'e1aaff51-0da2-2745-cf25-ab034aea8a25'
'OpaqueRef:85f25443-719e-105d-35aa-a1c1043fdd8e')))
[20150221T03:41:01.787Z|audit|xenserver1|13421 INET
0.0.0.0:80|task.add_to_other_config
D:43071305947a|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f'
'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API'
'task.add_to_other_config' (('self' 'Async.SR.create'
'577acdd7-2b8c-b177-bf45-f8422c4ed7ca'
'OpaqueRef:b933cd75-031b-6ff0-5a0a-2341507af648')))
[20150221T03:41:01.885Z|audit|xenserver1|14336|Async.SR.create
R:b933cd75031b|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f'
'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:SR_BACKEND_FAILURE_96: [ ;
The request is missing or has an incorrect target IQN parameter;
<?xml version=\"1.0\" ?> <iscsi-target-iqns> <TGT> <Index> 0 </Index>
<IPAddress> 192.168.10.1 </IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT>
<TGT> <Index> 1 </Index> <IPAddress> 192.168.11.1 </IPAddress>
<TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT> <Index> 2 </Index> <IPAddress> 192.168.20.1
</IPAddress>
<TargetIQN>iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT> <Index> 3 </Index> <IPAddress> 192.168.21.1
</IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT>
<TGT> <Index>
[20150221T03:41:02.883Z|audit|xenserver1|13421 INET
0.0.0.0:80|task.destroy D:5804968b2ed6|audit]
('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root'
'ALLOWED' 'OK' 'API' 'task.destroy' (('self' 'Async.SR.create'
'577acdd7-2b8c-b177-bf45-f8422c4ed7ca'
'OpaqueRef:b933cd75-031b-6ff0-5a0a-2341507af648')))
[20150221T03:41:06.598Z|audit|xenserver1|13421 INET
0.0.0.0:80|task.add_to_other_config
D:a09ea039fcd0|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f'
'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API'
'task.add_to_other_config' (('self' 'Async.SR.create'
'a0d8e973-9995-fb15-532e-f36ac5184c28'
'OpaqueRef:202cf910-0203-a943-9054-8d0e25a2b4a7')))
[20150221T03:41:06.638Z|audit|xenserver1|13421 INET
0.0.0.0:80|task.add_to_other_config
D:b4dd3f0d946a|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f'
'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API'
'task.add_to_other_config' (('self' 'Async.SR.create'
'a0d8e973-9995-fb15-532e-f36ac5184c28'
'OpaqueRef:202cf910-0203-a943-9054-8d0e25a2b4a7') ('value' ''
'e1aaff51-0da2-2745-cf25-ab034aea8a25'
'OpaqueRef:85f25443-719e-105d-35aa-a1c1043fdd8e')))
[20150221T03:41:06.679Z|audit|xenserver1|13421 INET
0.0.0.0:80|task.add_to_other_config
D:9b0a16a91401|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f'
'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API'
'task.add_to_other_config' (('self' 'Async.SR.create'
'a0d8e973-9995-fb15-532e-f36ac5184c28'
'OpaqueRef:202cf910-0203-a943-9054-8d0e25a2b4a7')))
[20150221T03:41:18.001Z|audit|xenserver1|14365 INET
0.0.0.0:80|handler:http/rrd_updates
D:3d1bf1a8b00c|audit] ('trackid=bf92de30818eb264b5673ffe734ad369'
'LOCAL_SUPERUSER' '' 'ALLOWED' 'OK' 'HTTP' 'http/rrd_updates' ())
[20150221T03:41:35.155Z|audit|xenserver1|14352|Async.SR.create
R:202cf9100203|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f'
'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:SR_BACKEND_FAILURE: [
non-zero exit; ; Traceback (most recent call last): File
\"/opt/xensource/sm/LVMoISCSISR\", line 582, in ?
SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File
\"/opt/xensource/sm/SRCommand.py\", line 343, in run sr =
driver(cmd, cmd.sr_uuid) File \"/opt/xensource/sm/SR.py\", line
142, in __init__ self.load(sr_uuid) File
\"/opt/xensource/sm/LVMoISCSISR\", line 156, in load self.iscsi =
self.iscsiSRs[0] IndexError: list index out of range ]'
'API' 'SR.create' (('host' 'xenserver1.local.iq.ufrj.br
<http://xenserver1.local.iq.ufrj.br/>'
'c8927969-b7bf-4a0f-9725-035550381d9c'
'OpaqueRef:03239cf1-76cf-9c86-b927-56ea27cb6b94') ('name_label'
'__gui__' '' '') ('name_description' 'SHOULD NEVER BE CREATED' '' '')))
[20150221T03:41:35.707Z|audit|xenserver1|13421 INET
0.0.0.0:80|task.destroy D:7ff3ff7db08a|audit]
('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root'
'ALLOWED' 'OK' 'API' 'task.destroy' (('self' 'Async.SR.create'
'a0d8e973-9995-fb15-532e-f36ac5184c28'
'OpaqueRef:202cf910-0203-a943-9054-8d0e25a2b4a7')))
***** /var/log/messages *****
Feb 21 01:41:01 xenserver1 xapi: [
info|xenserver1|14336|Async.SR.create R:b933cd75031b|dispatcher]
spawning a new thread to handle the current
task (trackid=8c8347a3f363fd40a9bd41eb1705050f)
Feb 21 01:41:01 xenserver1 xapi: [
info|xenserver1|14336|Async.SR.create R:b933cd75031b|storage_access]
SR d403d0fc-4a14-7f99-fe92-dfc6301ed764 will be implemented
by /services/SM/lvmoiscsi in VM
OpaqueRef:47f2c383-de3e-730e-9836-d652d5eae646
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|sm_exec
D:208b3f98f095|xapi] Session.create
trackid=70acc8c1d9618c45c2f07860af213730 pool=false uname=
originator= is_local_superuser=true auth_user_sid=
parent=trackid=9834f5af41c964e225f24279aefe4e49
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|sm_exec
D:208b3f98f095|xapi] Session.destroy
trackid=70acc8c1d9618c45c2f07860af213730
[error|xenserver1|14336|Async.SR.create
R:b933cd75031b|storage_access] Re-raising as SR_BACKEND_FAILURE_96 [
; The request is missing or has an incorrect target IQN parameter;
<?xml version="1.0" ?> <iscsi-target-iqns> <TGT> <Index> 0 </Index>
<IPAddress> 192.168.10.1 </IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT>
<TGT> <Index> 1 </Index> <IPAddress> 192.168.11.1 </IPAddress>
<TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT> <Index> 2 </Index> <IPAddress> 192.168.20.1
</IPAddress> <TargetIQN>
iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT>
<TGT> <Index> 3 </Index> <IPAddress> 192.168.21.1 </IPAddress>
<TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1
</TargetIQN> </TGT> <TGT> <Index> 4 </Index> <IPAddress> 192.168.30.1
</IPAddress> <TargetIQN>iqn.2015-01.b
Feb 21 01:41:06 xenserver1 xapi: [
info|xenserver1|14352|Async.SR.create R:202cf9100203|dispatcher]
spawning a new thread to handle the current
task (trackid=8c8347a3f363fd40a9bd41eb1705050f)
Feb 21 01:41:06 xenserver1 xapi: [
info|xenserver1|14352|Async.SR.create R:202cf9100203|storage_access]
SR 207535ee-c14c-7b48-3869-96318c1c7877 will be implemented
by /services/SM/lvmoiscsi in VM
OpaqueRef:47f2c383-de3e-730e-9836-d652d5eae646
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|sm_exec
D:4a4af7d945d9|xapi] Session.create
trackid=836c7784c6fd5a42124fb4bd64aa2cd5 pool=false uname=
originator= is_local_superuser=true auth_user_sid=
parent=trackid=9834f5af41c964e225f24279aefe4e49
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05155|bond|INFO|bond
bond0: shift 21kB of load (with hash 35) from eth1 to eth0 (now
carrying 233kB and 94kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05156|bond|INFO|bond
bond0: shift 224kB of load (with hash 39) from eth1 to eth0 (now
carrying 9kB and 318kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05157|bond|INFO|bond
bond0: shift 0kB of load (with hash 9) from eth0 to eth1 (now
carrying 318kB and 9kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05158|bond|INFO|bond
bond0: shift 1kB of load (with hash 12) from eth0 to eth1 (now
carrying 317kB and 10kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05159|bond|INFO|bond
bond0: shift 1kB of load (with hash 14) from eth0 to eth1 (now
carrying 316kB and 11kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05160|bond|INFO|bond
bond0: shift 0kB of load (with hash 20) from eth0 to eth1 (now
carrying 315kB and 11kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05161|bond|INFO|bond
bond0: shift 0kB of load (with hash 21) from eth0 to eth1 (now
carrying 315kB and 12kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05162|bond|INFO|bond
bond0: shift 0kB of load (with hash 25) from eth0 to eth1 (now
carrying 315kB and 12kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05163|bond|INFO|bond
bond0: shift 15kB of load (with hash 46) from eth0 to eth1 (now
carrying 300kB and 27kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05164|bond|INFO|bond
bond0: shift 2kB of load (with hash 70) from eth0 to eth1 (now
carrying 297kB and 30kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05165|bond|INFO|bond
bond0: shift 1kB of load (with hash 84) from eth0 to eth1 (now
carrying 295kB and 32kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05166|bond|INFO|bond
bond0: shift 0kB of load (with hash 94) from eth0 to eth1 (now
carrying 295kB and 32kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05167|bond|INFO|bond
bond0: shift 2kB of load (with hash 112) from eth0 to eth1 (now
carrying 293kB and 34kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05168|bond|INFO|bond
bond0: shift 28kB of load (with hash 118) from eth0 to eth1 (now
carrying 265kB and 62kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05169|bond|INFO|bond
bond0: shift 5kB of load (with hash 160) from eth0 to eth1 (now
carrying 259kB and 68kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05170|bond|INFO|bond
bond0: shift 5kB of load (with hash 171) from eth0 to eth1 (now
carrying 254kB and 73kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05171|bond|INFO|bond
bond0: shift 3kB of load (with hash 206) from eth0 to eth1 (now
carrying 251kB and 76kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05172|bond|INFO|bond
bond0: shift 2kB of load (with hash 254) from eth0 to eth1 (now
carrying 249kB and 78kB load, respectively)
ovs|05173|ofproto|INFO|xapi0: 29 flow_mods in the last 59 s (29 adds)
Feb 21 01:41:35 xenserver1 fe: 2498 (/opt/xensource/sm/LVMoISCSISR
<methodCall><methodName>sr_create</methodName><...) exited with code 1
Feb 21 01:41:35 xenserver1 xapi: [ info|xenserver1|14352|sm_exec
D:4a4af7d945d9|xapi] Session.destroy
trackid=836c7784c6fd5a42124fb4bd64aa2cd5
[error|xenserver1|14352|Async.SR.create
R:202cf9100203|storage_access] Re-raising as SR_BACKEND_FAILURE [
non-zero exit; ; Traceback (most recent call last): File
"/opt/xensource/sm/LVMoISCSISR", line 582, in ?
SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File
"/opt/xensource/sm/SRCommand.py", line 343, in run sr
= driver(cmd, cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line
142, in __init__ self.load(sr_uuid) File
"/opt/xensource/sm/LVMoISCSISR", line 156, in load self.iscsi
= self.iscsiSRs[0] IndexError: list index out of range ]
Feb 21 01:41:56 xenserver1 xenstored: D3.14 rm attr/eth0
Feb 21 01:41:56 xenserver1 xenstored: D3 write attr/eth0/ip
10.7.0.243
Feb 21 01:41:56 xenserver1 xenstored: D3 write
attr/eth0/ipv6/0/addr fe80::484b:acff:fe91:5b19
***** /var/log/SMlog *****
Feb 21 01:41:01 xenserver1 SM: [2444] _testHost: Testing host/port: 192.168.10.1,3260
Feb 21 01:41:01 xenserver1 SM: [2444] lock: opening lock file
/var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: acquired
/var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: released
/var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: closed
/var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: opening lock file
/var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: acquired
/var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: released
/var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: closed
/var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] Raising exception [96, The
request is missing or has an incorrect target IQN parameter]
Feb 21 01:41:06 xenserver1 SM: [2498] lock: opening lock file
/var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: acquired
/var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: released
/var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] Raising exception [202, General
backend error [opterr=rc: 21, stdout: , stderr: iscsiadm: No active
sessions.
Feb 21 01:41:06 xenserver1 SM: [2498] ]]
Feb 21 01:41:06 xenserver1 SM: [2498] ['iscsiadm', '-m', 'session']
failed with (u'General backend error [opterr=rc: 21, stdout: ,
stderr: iscsiadm: No active sessions.\n]',)
Feb 21 01:41:06 xenserver1 SM: [2498] lock: closed
/var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] ['ls', '/sys/class/scsi_host',
'-1', '--color=never']
Feb 21 01:41:06 xenserver1 SM: [2498] pread SUCCESS
Feb 21 01:41:06 xenserver1 SM: [2498] []
Feb 21 01:41:06 xenserver1 SM: [2498] PATHDICT: key
192.168.10.1:3260: {'path': '/dev/iscsi/*/192.168.10.1:3260',
'ipaddr': '192.168.10.1', 'port': 3260L}
Feb 21 01:41:06 xenserver1 SM: [2498] lock: opening lock file
/var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: acquired
/var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: released
/var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: closed
/var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] Discovery for IP 192.168.10.1
returned [('192.168.10.1:3260', '2',
'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'),
('192.168.11.1:3260', '2',
'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'),
('192.168.20.1:3260', '2',
'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'),
('192.168.21.1:3260', '2',
'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'),
('192.168.30.1:3260', '2',
'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'),
('192.168.31.1:3260', '2',
'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1')]
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.10.1,3260
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.11.1,3260
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.20.1,3260
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Connect failed after
2 seconds (192.168.20.1) - (113, 'No route to host')
Feb 21 01:41:06 xenserver1 SM: [2498] Raising exception [141, Unable
to connect to ISCSI service on target]
(192.168.20.1:3260)
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.21.1,3260
Feb 21 01:41:07 xenserver1 SM: [2498] _testHost: Connect failed after
2 seconds (192.168.21.1) - (113, 'No route to host')
Feb 21 01:41:07 xenserver1 SM: [2498] Raising exception [141, Unable
to connect to ISCSI service on target]
(192.168.21.1:3260)
Feb 21 01:41:07 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.30.1,3260
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Connect failed after
2 seconds (192.168.30.1) - timed out
Feb 21 01:41:09 xenserver1 SM: [2498] Raising exception [141, Unable
to connect to ISCSI service on target]
(192.168.30.1:3260)
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.31.1,3260
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Connect failed after
2 seconds (192.168.31.1) - (113, 'No route to host')
Feb 21 01:41:09 xenserver1 SM: [2498] Raising exception [141, Unable
to connect to ISCSI service on target]
(192.168.31.1:3260)
Feb 21 01:41:09 xenserver1 SM: [2498] lock: opening lock file
/var/lock/sm/iscsiadm/running
Feb 21 01:41:09 xenserver1 SM: [2498] lock: acquired
/var/lock/sm/iscsiadm/running
Feb 21 01:41:09 xenserver1 SM: [2498] lock: released
/var/lock/sm/iscsiadm/running
Feb 21 01:41:09 xenserver1 SM: [2498] lock: closed
/var/lock/sm/iscsiadm/running
Feb 21 01:41:09 xenserver1 SM: [2498] Discovery for IP 192.168.11.1
returned [('192.168.10.1:3260', '2',
'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'),
('192.168.11.1:3260', '2',
'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'),
('192.168.20.1:3260', '2',
'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'),
('192.168.21.1:3260', '2',
'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'),
('192.168.30.1:3260', '2',
'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'),
('192.168.31.1:3260', '2',
'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1')]
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.10.1,3260
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.11.1,3260
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.20.1,3260
Feb 21 01:41:11 xenserver1 SM: [2498] _testHost: Connect failed after
2 seconds (192.168.20.1) - timed out
Feb 21 01:41:11 xenserver1 SM: [2498] Raising exception [141, Unable
to connect to ISCSI service on target]
(192.168.20.1:3260)
Feb 21 01:41:11 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.21.1,3260
Feb 21 01:41:11 xenserver1 SM: [2498] _testHost: Connect failed after
2 seconds (192.168.21.1) - (113, 'No route to host')
Feb 21 01:41:11 xenserver1 SM: [2498] Raising exception [141, Unable
to connect to ISCSI service on target]
(192.168.21.1:3260)
Feb 21 01:41:11 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.30.1,3260
Feb 21 01:41:12 xenserver1 SM: [2498] _testHost: Connect failed after
2 seconds (192.168.30.1) - (113, 'No route to host')
Feb 21 01:41:12 xenserver1 SM: [2498] Raising exception [141, Unable
to connect to ISCSI service on target]
(192.168.30.1:3260)
Feb 21 01:41:12 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.31.1,3260
Feb 21 01:41:14 xenserver1 SM: [2498] _testHost: Connect failed after
2 seconds (192.168.31.1) - timed out
Feb 21 01:41:14 xenserver1 SM: [2498] Raising exception [141, Unable
to connect to ISCSI service on target]
(192.168.31.1:3260)
Feb 21 01:41:15 xenserver1 SM: [2498] lock: opening lock file
/var/lock/sm/iscsiadm/running
Feb 21 01:41:15 xenserver1 SM: [2498] lock: acquired
/var/lock/sm/iscsiadm/running
Feb 21 01:41:35 xenserver1 SM: [2498] lock: released
/var/lock/sm/iscsiadm/running
Feb 21 01:41:35 xenserver1 SM: [2498] Raising exception [202, General
backend error [opterr=rc: 21, stdout: , stderr: iscsiadm: cannot make
connection to 192.168.20.1: No route to host
Feb 21 01:41:35 xenserver1 SM: [2498] iscsiadm: cannot make
connection to 192.168.20.1: No route to host
Feb 21 01:41:35 xenserver1 last message repeated 3 times
Feb 21 01:41:35 xenserver1 SM: [2498] iscsiadm: connect to
192.168.20.1 timed out
Feb 21 01:41:35 xenserver1 SM: [2498] iscsiadm: connection login
retries (reopen_max) 5 exceeded
Feb 21 01:41:35 xenserver1 SM: [2498] iscsiadm: No portals found
Feb 21 01:41:35 xenserver1 SM: [2498] ]]
Feb 21 01:41:35 xenserver1 SM: [2498] Raising exception [68, ISCSI
login failed, verify CHAP credentials]
Feb 21 01:41:35 xenserver1 SM: [2498] lock: closed
/var/lock/sm/iscsiadm/running
EXCEPTION SR.SROSError, ISCSI login failed, verify CHAP credentials
Feb 21 01:41:35 xenserver1 SM: [2498] File
"/opt/xensource/sm/LVMoISCSISR", line 119, in load
Feb 21 01:41:35 xenserver1 SM: [2498] map =
iscsilib.discovery(tgt_ip,iscsi.port,iscsi.chapuser,iscsi.chappassword,targetIQN=IQN)
Feb 21 01:41:35 xenserver1 SM: [2498] File
"/opt/xensource/sm/iscsilib.py", line 156, in discovery
Feb 21 01:41:35 xenserver1 SM: [2498] raise
xs_errors.XenError('ISCSILogin')
Feb 21 01:41:35 xenserver1 SM: [2498] File
"/opt/xensource/sm/xs_errors.py", line 52, in __init__
Feb 21 01:41:35 xenserver1 SM: [2498] raise
SR.SROSError(errorcode, errormessage)
Feb 21 01:41:35 xenserver1 SM: [2498]
Feb 21 01:41:54 xenserver1 updatempppathd: [7617] The garbage
collection routine returned: 0
Post by Tobias Kreidl
When I mentioned running the SrRepairAction procedure, I meant in
conjunction with what is/not/happening in_XenCenter_as a follow-up
to fix things.
So then, it would seem the piece remaining to resolve is what is
missing in XenCenter that is not triggered. My guess is that it can
perhaps only perform a connection using one IP address (or
wildcarded address) and hence apparently doesn't understand what to
do with anything involving a multiSession/multihome list?
It would be a useful diagnostic to see what is parsed out and
processed if you pass in the whole multihome list of
192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260
to XenCenter and see what it ends up trying to do with it.
-=Tobias
Post by Vinícius Ferrão
Guys, another important thing that I've observed.
All the iscsiadm and multipath commands are useless. I've attached the same storage just issuing a specific PBD command from the Pool Master. So this simplifies the problem a lot.
xe pbd-create \
sr-uuid=f8d481b8-be10-922e-b875-91c4d73d341d \
host-uuid=44955863-f4a3-4770-bd7a-f4342dd7925a \
device-config:multiSession="192.168.30.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.31.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" \
device-config:target=192.168.30.1,192.168.31.1 \
device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 \
device-config:targetIQN=* \
device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 \
device-config:port=3260
The device-config attributes are sufficient to create all the connection.
Thanks,
Vinicius.
It may be possible to at least temporarily get this work by forcing the SrRepairAction procedure to be run.
I didn't have time yet, but wanted to compare what it does in sequence vs. what is done by default in the xapi_sr
program to see what is different. As far as XenCenter is concerned, perhaps if multiple IP target addresses are entered in the dialog box (as opposed to a single one or a wildcard entry), then a different means could be triggered to ensure that each host does a proper PBD creation and plug-in for that new SR.
-=Tobias
Post by Vinícius Ferrão
Dave,
I've added the "future" server to our pool. And happened exactly what you said. The configurations were cloned. But obviously it failed to clone the PDB's and the Network configuration for the iSCSI network.
1. The network configuration on added host should be "confirmed" to the user, and not done automatically, so the user can adapt is to specific needs.
2. After this the PDB creation should be done accordingly. If the network topology has changed by the user, so the PDBs must be created with this new topology.
In resume, I've three host at this moment on the pool. So I can do the HA tests once again, and I will report to the list the results. At this moment the only thing that failed is cloning the iSCSI network configurations and plugging the PDBs. Only the bounded NIC0+NIC1 was created on the added host.
Thanks in advance,
Vinícius.
Post by Dave Scott
Regarding Point 3) I believe what is meant is that you should test having more than one SR that you are connecting to using the same IP addresses for the target array. You know it works fine to create one SR, so can you add additional ones the same way and have them all live together in harmony? :-)
As to the sr-create command, the host-uuid is an optional parameter and hence does not need to be named for the pool master or any other. The only required parameters are the name-label and the SR type. Have you tried that to see if the behavior is any different compared to when you do supply a host name?
I believe the underlying Xapi data model can express what you want, but there are 1 or 2 places where Xapi will helpfully create PBDs for you by cloning the master, which you just need to be aware of.
For example you should be able to use something like
$ xe sr-create .. host-uuid=<any host> shared=false
Setting “shared=false” initially should avoid Xapi ‘helpfully’ pre-creating identical PBDs for every host with the same configuration[1]. You should end up with an SR attached to a single host.
$ xe sr-param-set uuid= shared=true
$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...
$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...
[ as an aside, it’s obviously not great that the PBDs you get depend on exactly when you declare the SR is shared. This has happened because the original design had all PBDs being distinct, and the notion of ‘copying the master’ was added relatively late ]
Whenever a new host is added to the pool, Xapi will clone the master’s PBD for you— you would have to destroy it and re-create it. I suppose cloning the master’s configuration is as good a guess as any in this situation :-)
Dave
[1]https://github.com/xapi-project/xen-api/blob/b6f28c52fd9726553968b410e6b8cced5401f385/ocaml/xapi/xapi_sr.ml#L209
-=Tobias
Post by Vinícius Ferrão
Hello guys,
Answering about all the questions and issues. For those who aren't understanding the network topology, I've made a drawing of the solution. The image is here:https://www.dropbox.com/s/6zwerk6igcyqou3/UFRJ-IQ-VirtualPool.jpg?dl=0
192.168.10.1
192.168.11.1
192.168.20.1
192.168.21.1
The convention to this numbering scheme was: 192.168.{XenServerNumber}{InterfaceNumber}.{Storage(1)/Host(2)}.
192.168.10.2 (XenServer #1 - Interface #0)
192.168.11.2 (XenServer #1 - Interface #1)
192.168.20.2 (XenServer #2 - Interface #0)
192.168.21.2 (XenServer #2 - Interface #1)
Hope this completely clarifies the architecture/topology of the iSCSI network.
Restarting both hosts on the same time. [OK]
Restarting the second host. [OK]
Restarting the pool master. [OK]
2. HA is working too. It fences only the damaged host. I even tried a VM Failover. HA happened without any problem and the lost VM has been restarted on other server. To do the HA test, I've power cycled the host, simulated a hardware crash scenario. A cool picture:https://www.dropbox.com/s/g7h7rljk8vh97rg/Screenshot%202015-02-18%2015.45.41.png?dl=0
3. I haven't understood this test. You are asking to create a new LUN and add a new Shared Repository? It's not clear what you're saying with "independent SR".
4. Storage XenMotion works too. I've moved a running VM from local storage (on the Host #2) to the iSCSI pool without any problem. The reverse process works too.
5. I will test this tomorrow, when I will got physical access to the servers (carnival holiday here in BR). But about this specific test, I can't see any problem with plain Xen Motion, since the data is copied over the management network and not the iSCSI network. But I can see a good point when dealing with Storage XenMotion.
Finally about the changes on XenCenter or XenServer/XAPI itself. One thing to prove this point is to create the SR via the CLI. If I would be able to do this, the problem should definitely be on XenCenter. As I said before, I've created a broken Software iSCSI SR via XenCenter and them mapped the iSCSI interfaces, enabled multipath and created the PBD over the CLI.
The iSCSI and Multipath should be done in the physical host. But the PBD creation can be done without any problem on the Pool Master.
device-config: name-label= shared= type=
In this case sm-config and device-config are the problematic ones. Assuming that the host-uuid is always the pool master.
Thanks in advance,
Vinícius.
I’ll add in a few more tests….
1. Host restart. Does the normal PBD plug mechanism work with such a configuration
2. HA. While HA with two hosts isn’t a supported configuration, in this model it might work properly. If you remove the network cables for a single host, does it fence, or do both hosts fence?
3. Multiple SRs. If you have multiple LUNs on the FreeNAS exported and then used as independent SRs, does everything work as it should?
4. Storage XenMotion. With a vdi on one SR presented with the FreeNAS, copy it to any another storage option
5. Motion during path failure. With one path down on a given host, does XenMotion and Storage XenMotion still function correctly?
For those who are questioning the value of such a configuration, this is in essence a “cluster in a box” solution. These configurations were very popular 10-15 years ago as highly available compute infrastructure was quite expensive back then. Today we can accomplish the same things we did back then using virtualization, but “cluster in a box” can always be valuable for those with limited IT skills or for smaller organizations wanting least cost with fewest points of failure.
-tim
Sent: Monday, February 16, 2015 7:24 PM
To: Tobias Kreidl
Subject: Re: [xs-devel] Switchless shared iSCSI Network with /30 links
Tobias,
I've done more tests about the SR creation over XenCenter.
All the process definitely occurs only on the pool master. Even if the specified networks are not part of the networks of the host. The connection is tried on the default route.
192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1
The target IQN is discovered without any problem, since the iSCSI Storage reports the LUNs on any interface.
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 192.168.10.2:55870 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52654 192.168.11.1:3260 TIME_WAIT
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.11.2:52656 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
******************
tcp 0 1 10.7.0.101:56481 192.168.21.1:3260 SYN_SENT
******************
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
******************
tcp 0 1 10.7.0.101:52787 192.168.30.1:3260 SYN_SENT
******************
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
If you're wondering the 192.168.30.1 is reported because in the future we will add another XS host to the pool. The point here is that XenCenter should not lookup for this network since it isn't specified during the Software iSCSI creation. On my understanding it should only lookup for the specified addresses.
tcp 0 48 10.7.0.102:22 192.168.172.1:58046 ESTABLISHED
udp 0 0 192.168.21.2:123 0.0.0.0:*
udp 0 0 192.168.20.2:123 0.0.0.0:*
As you said, perhaps propagating the initial connection through the host will solve the problem without modifying the XenCenter interface. A more sophisticated approach would be query the network-list over XAPI, retrieve the PIFs associated to those networks and them just check the IP addresses on the SR creation request and match them with the equivalent PIFs.
uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa
name-label ( RW): iSCSI #0
PIF-uuids (SRO): 3baa8436-feca-cd04-3173-5002d9ffc66b; 362d63cb-647a-465f-6b81-1b59cd3b1f5d
MTU ( RW): 9000
bridge ( RO): xenbr2
other-config (MRW): automatic: true
default-locking-mode ( RW): unlocked
uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5c:64
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
host-name-label ( RO):xenserver2.local.iq.ufrj.br <http://xenserver2.local.iq.ufrj.br/>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.20.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5a:1c
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-slave-of ( RO): <not in database>
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c
host-name-label ( RO):xenserver1.local.iq.ufrj.br <http://xenserver1.local.iq.ufrj.br/>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.10.2
netmask ( RO): 255.255.255.252
IPv6-configuration-mode ( RO): None
primary-address-type ( RO): IPv4
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
So as you can see, we have the IP addresses and the network mask. Which is sufficient to an efficient and precise algorithm of SR creation.
I thinks this clarifies everything we discussed until now. A patch on XenCenter appears to be really viable.
Many thanks,
Vinicius.
Yes, correct. The MD36xx looks like two separate controller units, each acting as a separate device. For a standalone server or some storage devices, that's right that you need a separate subnet for each connection.
Because a single connection isn't seeing the broadcast from the other connectors at the time of the initial connection from one host, the pool is unaware of the others. The "repair" operation causes this to be conducted from each host, and as I guessed, then fixes the SR connectivity for the entire pool. I am speculating that the request for a new connection of this type via XenCenter could perhaps be solved by propagating that initial connection process to the rest of the hosts in the pool and then rescan the SR.
-=Tobias
Hi Tobias,
As for I know, the Dell MD36xx is a dual controller based storage. So it's basically "two computers". Considering this with the added knowledge that I've about TCP networking there's a golden rule that says: only one IP address on a given subnet on a single computer. So I can't use the same network on my case. That's why I use point-to-point links too.
Since I've only one machine acting as a storage server (It's a Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do need four separate networks. FreeNAS is FreeBSD based, and BSD enforces this rule by default. I can't have two IP addresses of the same subnet on same machine, even if I manually configure it.
If this wasn't bad TCP network practice, I would be able to configure the SR via XenCenter, since it will "find" the same network on the others hosts without problem.
About the XenCenter, it really appears to be only a missing function on the software. If I was a good programmer I would try to implement this. But this is beyond my knowledge... :(
Thanks once again,
Vinícius.
No reason to necessarily use four separate subnets. You could have 192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and 2.2 on the other controller (on, for example, an MD36xx that is standard). If however it is something like an MD32xx which recommends four separate subets, then of course, use four. One should follow the vendor's specifications.
I will respond in more detail later when I get a chance, Vinícius, but after a quick read, I agree with your findings from your other email. It would appear that the initial connection is indeed evidently a missing function in XenCenter and perhaps all that is needed to make this work "out of the box". As to the wildcard discovery, it's generally preferred as it allows the host to figure out the path on its own. The only reason I could see not using a wildcard discovery would be if there were any chance for overlaps and ambiguous connections.
Regards,
--Tobias
Hello Dave,
+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+
192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30
192.168.10.2/30
192.168.11.2/30
192.168.20.2/30
192.168.21.2/30
That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0
Cheers,
Vinícius.
Hi,
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can’t understand why it isn’t a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I’m trying to do is achieve the same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.
Perhaps we aren’t agreeing at some points due to different markets. For example, I’ve used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked… Wasn’t for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn’t the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.
Leaving all those “political” points aside, let’s talk technically :)
I’ve managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this:https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0
I’m really excited because it worked. So let’s me explain what I’ve did.
First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP’s of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD’s are the physical connect from separate XS hosts to the shared storage. So it’s not related to other hosts, it’s exclusively to the host machine that I’m using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I’d like to understand your configuration better— have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?
Thanks,
Dave
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
Pics:https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0
I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It’s just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It’s almost 4AM here, and I was really anxious to report this in the list.
Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it’s only a interface to control some policies.
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/
It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn’t apply.
If I understood correctly the SR is created with the appropriate PBD’s. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD’s from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I’m opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that’s a very common architecture and VMware is doing this on small pools! So let’s do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.
Germano Percossi
2015-03-30 10:58:20 UTC
Permalink
Post by Tobias Kreidl
Perhaps another place to post this would be also on the Xen development
https://xenorg.uservoice.com/forums/172169-xen-development as typically
a lot of work gets done and implemented there before it is absorbed by
XenServer itself.
Hi Tobias,

While what you say is somehow true for some components, this is surely
not true for the management stack (and blktap/tapdisk to mention another).
Most of the stuff, in this case, happen in the xapi-project on GitHub.

Germano

Alex Brett
2015-02-18 18:15:21 UTC
Permalink
2. HA.  While HA with two hosts isn’t a supported configuration, in this model it might work properly.  If you remove the network cables for a single host, does it fence, or do both hosts fence?
This won’t help the issues with using HA with two hosts – the issue that occurs here is described in http://support.citrix.com/article/CTX129721 - in summary with HA in some situations a host decides whether it should survive or not by whether it is part of the largest partition. With a failure in a two host pool you obviously always end up with two equal sized partitions (one containing the ‘live’ host, and one the ‘dead’ host) – in the event of equal size partitions a deterministic tie-breaking solution is used to decide which one survives, the current implementation being that the host with the lowest host UUID survives. As such if the host that failed was the one with the lower host UUID, the other host will also fence.

It's worth noting this can mean if you only test failing one host that all can appear to work correctly, as if you happen to pick the host with the higher UUID the other one will survive as expected, but if you reverse the test and fail the o
Tobias Kreidl
2015-02-18 18:19:13 UTC
Permalink
Yes. I'm surprised this still hasn't been fixed, as HA-Lizard can
purportedly address this "split brain" issue just fine: http://halizard.com/
Post by Alex Brett
2. HA. While HA with two hosts isn’t a supported configuration, in this model it might work properly. If you remove the network cables for a single host, does it fence, or do both hosts fence?
This won’t help the issues with using HA with two hosts – the issue that occurs here is described in http://support.citrix.com/article/CTX129721 - in summary with HA in some situations a host decides whether it should survive or not by whether it is part of the largest partition. With a failure in a two host pool you obviously always end up with two equal sized partitions (one containing the ‘live’ host, and one the ‘dead’ host) – in the event of equal size partitions a deterministic tie-breaking solution is used to decide which one survives, the current implementation being that the host with the lowest host UUID survives. As such if the host that failed was the one with the lower host UUID, the other host will also fence.
It's worth noting this can mean if you only test failing one host that all can appear to work correctly, as if you happen to pick the host with the higher UUID the other one will survive as expected, but if you reverse the test and fail the other host you'll find problems...
Alex
Loading...